News ChatGPT's New Code Interpreter Has Giant Security Hole, Allows Hackers to Steal Your Data

[Admin]

A new ChatGPT feature lets you upload files full of data or code, but it also makes them vulnerable to exfiltration.

ChatGPT's New Code Interpreter Has Giant Security Hole, Allows Hackers to Steal Your Data : Read more

 

[vanadiel007]

Maybe they should use AI to find the security holes in their own code. It's pretty amazing how dumb AI is, if it cannot find or see security holes in it's own interface.

 

[JamesJones44]

Maybe they should use AI to find the security holes in their own code. It's pretty amazing how dumb AI is, if it cannot find or see security holes in it's own interface.

The word AI is grossly over stated for what is available today. Machine learning is a much better fit, but not catchy enough for marketers.

I've always wondered if Alan Turing would change the Turing test if he were alive today or if he would say this is what he meant.

 

[bit_user]

The word AI is grossly over stated for what is available today. Machine learning is a much better fit, but not catchy enough for marketers.

I think we're past that point. LLMs can deduce, abstract, reason, and synthesize. Although they're far from perfect, they're well beyond simple machine learning.

I've always wondered if Alan Turing would change the Turing test if he were alive today or if he would say this is what he meant.

I think he'd be amazed by what LLMs can do. Don't forget that the Turing Test had stymied conventional solutions for some 8 decades, before LLMs came onto the scene. Sure, we need new benchmarks, but being beaten doesn't make the Turing Test a bad or invalid benchmark. Heck, the movie Blade Runner already anticipated the need for something beyond the vanilla Turing Test - and that was over 40 years ago!

 

[FoxtrotMichael-1]

I think we're past that point. LLMs can deduce, abstract, reason, and synthesize. Although they're far from perfect, they're well beyond simple machine learning.

I actually registered here just to reply to this comment. As someone who works with LLMs every single day professionally: they absolutely cannot deduce or reason. This is an extremely common misconception that’s incredibly dangerous. The LLM, regardless of how advanced it is, has absolutely no clue what it’s actually saying. LLMs essentially do complex token prediction analysis to generate strings that sound and read like human language. An LLM can imitate deduction and reason if it has been trained on speech that sounds as if it is deducing and reasoning. That’s all. Believe that it can actually deduce and reason at your own risk - you’ll be listening to a voice that has no idea what it’s actually saying.

 

[COLGeek]

I actually registered here just to reply to this comment. As someone who works with LLMs every single day professionally: they absolutely cannot deduce or reason. This is an extremely common misconception that’s incredibly dangerous. The LLM, regardless of how advanced it is, has absolutely no clue what it’s actually saying. LLMs essentially do complex token prediction analysis to generate strings that sound and read like human language. An LLM can imitate deduction and reason if it has been trained on speech that sounds as if it is deducing and reasoning. That’s all. Believe that it can actually deduce and reason at your own risk - you’ll be listening to a voice that has no idea what it’s actually saying.

Well said. Thanks for sharing.

 

[Order 66]

I actually registered here just to reply to this comment. As someone who works with LLMs every single day professionally: they absolutely cannot deduce or reason. This is an extremely common misconception that’s incredibly dangerous. The LLM, regardless of how advanced it is, has absolutely no clue what it’s actually saying. LLMs essentially do complex token prediction analysis to generate strings that sound and read like human language. An LLM can imitate deduction and reason if it has been trained on speech that sounds as if it is deducing and reasoning. That’s all. Believe that it can actually deduce and reason at your own risk - you’ll be listening to a voice that has no idea what it’s actually saying.

I didn't know that, I just thought that they were actually capable of deduction. The more you know I guess.

 

[baboma]

>I actually registered here just to reply to this comment. As someone who works with LLMs every single day professionally: they absolutely cannot deduce or reason.

In a practical sense, it doesn't matter (that LLMs can't reason), as along as they can give answers to queries that emulate reasoning/deduction to provide a satisfactory response--in lieu of no response at all.

LLM critics harp on pendantic wordplay, arguing that LLMs aren't "real AI" or that they're "autocompletes on steroids" or whatever. They're missing the point, which isn't what LLM is or isn't, but what it can do. Nobody cares if LLM is or isn't "real AI". There's a reason that companies are pumping multiples of billions into LLM/AI. Your job is likely a direct consequence of that trend. It will impact many more jobs, and many more industries.

 

[FoxtrotMichael-1]

>I actually registered here just to reply to this comment. As someone who works with LLMs every single day professionally: they absolutely cannot deduce or reason.

In a practical sense, it doesn't matter (that LLMs can't reason), as along as they can give answers to queries that emulate reasoning/deduction to provide a satisfactory response--in lieu of no response at all.

LLM critics harp on pendantic wordplay, arguing that LLMs aren't "real AI" or that they're "autocompletes on steroids" or whatever. They're missing the point, which isn't what LLM is or isn't, but what it can do. Nobody cares if LLM is or isn't "real AI". There's a reason that companies are pumping multiples of billions into LLM/AI. Your job is likely a direct consequence of that trend. It will impact many more jobs, and many more industries.

Call it being pedantic if you like but reason and imitation of reason are two totally different things. I'm actually a huge LLM proponent so I'm not trying to downplay how useful of a tool they are or how efficient they can help us become. The real problem is going to come when people expect LLMs to make decisions. To some extent, this is already happening (see Microsoft Security Copilot). When someone needs a decision to be made I'd personally much rather have someone who can actually reason and deduce making the call and not an LLM that can simply sound like it is but has no idea what's going on in any case.

 

[JamesJones44]

I think we're past that point. LLMs can deduce, abstract, reason, and synthesize. Although they're far from perfect, they're well beyond simple machine learning.

I agree with that to a degree but it's still centered around trained data inference, advanced inference, but still inference. It doesn't think about what it's doing, it runs an algorithm against data and adjusts the weights based on responses. IMO that's still machine learning. If self learning every comes into play then I would agree we are passed machine learning.

 

[bit_user]

I actually registered here just to reply to this comment.

Welcome! Thanks for sharing your perspective. Hopefully, you'll find a few reasons to stick around.

I actually registered here just to reply to this comment. As someone who works with LLMs every single day professionally: they absolutely cannot deduce or reason.

I think it's dangerous to be too dismissive.

Do you design LLMs, or is your professional capacity entirely on the usage end, or perhaps training? If you're going to claim authority, then I think it's fair that we ask you to specify the depth of your expertise and length of your experience. There are now lots of people who use LLMs in their jobs, yet still very few who actually design them and understand how they actually work.

LLMs essentially do complex token prediction analysis to generate strings that sound and read like human language.

Yes, that's the training methodology, but it turns out that you need to understand a lot about what's being said in order to be good at predicting the next word.

Try it some time: take a paper from a medical or some other scientific journal, in a field where you have no prior background, and see how well you can do at predicting each word, based on the prior ones.

An LLM can imitate deduction and reason if it has been trained on speech that sounds as if it is deducing and reasoning.

At some point, an imitation becomes good enough to be indistinguishable from the real deal. At that point, is the distinction even meaningful?

Believe that it can actually deduce and reason at your own risk - you’ll be listening to a voice that has no idea what it’s actually saying.

This is sort of overly-anthropomorphizing it. If I fit a mathematical model to a regular physical process, I can get accurate predictions without the model "understanding" physics.

AI doesn't need metacognition in order to perform complex cognitive tasks. When we see smart animals do things like solving puzzles, we don't make our acceptance of what we see contingent on them "understanding what they're doing".

 

[baboma]

>The real problem is going to come when people expect LLMs to make decisions.

That's not reality. Reality is this: People adapt to the tool's utility. We know that LLMs can give bad responses, but it can give good responses most of the time. Therefore we use the tool, but for anything substantive, we double check. So, I disagree with your blanket "people expect LLMs to make decisions" comment. What you're saying is that people are stupid, and in the main, people aren't stupid.

>To some extent, this is already happening (see Microsoft Security Copilot).

Sure, it happened with the Humane demo. It's called a gaffe. The entire industry is still in beta. The might of trillions of dollars are directed toward making LLMs better, which means mimimizing or eliminating hallucinations altogether.

Regardless, this is irrelevant to the pendantic "real or fake" argument. LLMs can be real thinkers and they can still lie, and if that's the case, people should be a lot more worried. So, good that LLMs can't really think.

 

[bit_user]

The real problem is going to come when people expect LLMs to make decisions. To some extent, this is already happening (see Microsoft Security Copilot).

If you want an AI that's good at making decisions, then it needs to be trained that way. Similarly, if you want one that's good at math, you need to give it a sufficiently comprehensive training set.

The same would be true of a human. We don't expect a student to be good at math without the proper instruction and coursework.

When someone needs a decision to be made I'd personally much rather have someone who can actually reason and deduce making the call and not an LLM that can simply sound like it is but has no idea what's going on in any case.

Humans have an annoying tendency to do ex post facto reasoning. fitting a logical narrative to a set of actions or decisions. If you treat a human as the gold standard, you probably find out what you're dealing with is actually iron pyrite.

 

[bit_user]

I agree with that to a degree but it's still centered around trained data inference, advanced inference, but still inference. It doesn't think about what it's doing, it runs an algorithm against data and adjusts the weights based on responses. IMO that's still machine learning. If self learning every comes into play then I would agree we are passed machine learning.

The error I think you and others are making is in assuming that certain cognitive processes can only be conducted by something that's "truly intelligent". Then, you guys are saying "I know it's not truly intelligent, because it lacks the degree of self-awareness it should have if it were".

Stop conflating certain cognitive capabilities with "true intelligence". If you want to know whether LLMs can do a certain cognitive task, devise a set of fairly easy tests they couldn't pass without said capability. Set aside your preconceptions about intelligence and just look at how it performs on those tests.

 

[jakezz]

I actually registered here just to reply to this comment. As someone who works with LLMs every single day professionally: they absolutely cannot deduce or reason. This is an extremely common misconception that’s incredibly dangerous. The LLM, regardless of how advanced it is, has absolutely no clue what it’s actually saying. LLMs essentially do complex token prediction analysis to generate strings that sound and read like human language. An LLM can imitate deduction and reason if it has been trained on speech that sounds as if it is deducing and reasoning. That’s all. Believe that it can actually deduce and reason at your own risk - you’ll be listening to a voice that has no idea what it’s actually saying.

What does it matter if it has no idea what it's saying if it can so masterfully elicit a response from a human better than another human can (ex: phishing techniques). I see this argument a lot, that it is only coming up with the next word or token by probability, but, isn't that the same thing humans do? When you really think about it, don't humans learn speech much of the same way a lot of these AI do? Doesn't a human with increasing intellect speak more proper and have more structure and rules in their speech, form words into common speech patterns or phrasing for clearer communication, making it more predictable? The biggest difference I can see is that AI is able to communicate with a much larger range of humans with seemingly no language barriers. And receive, process, and give a response at a much faster rate than a human ever will. And this is all just the beginning...

Coincidentally I actually also registered just to reply to your comment lol..

 

[FoxtrotMichael-1]

Do you design LLMs, or is your professional capacity entirely on the usage end, or perhaps training? If you're going to claim authority, then I think it's fair that we ask you to specify the depth of your expertise and length of your experience.

This is a perfectly reasonable question and I didn't intend to misrepresent my actual experience with LLMs or ML in general. I do not design or build LLMs. I integrate LLMs into other systems; usually, this means working with APIs and prompt engineering. However, just because I consume LLMs doesn't mean that I don't understand how they work on a functional level. I've written some of the foundational algorithms of machine learning from scratch and I'm fairly familiar with convolutional neural networks.

This is sort of overly-anthropomorphizing it. If I fit a mathematical model to a regular physical process, I can get accurate predictions without the model "understanding" physics.

If you fit a mathematical model to a regular physical process to get accurate predictions, you've created an advanced calculator, not artificial intelligence.

At some point, an imitation becomes good enough to be indistinguishable from the real deal. At that point, is the distinction even meaningful?

This and other statements get to the bottom of our disagreement. I'm classifying things by their essence while you want to classify them by their utility. Some will call my side of the argument "pedantic" but I'm not convinced it is. Let's consider a simple thought experiment: if everyone in your life (friends, family, etc) were replaced by indistinguishable automatons would it matter to you? If it wouldn't bother you, then at least you're being academically consistent, although this brings up serious concerns about how you would view the value of any human properly.

 

[JamesJones44]

The error I think you and others are making is in assuming that certain cognitive processes can only be conducted by something that's "truly intelligent". Then, you guys are saying "I know it's not truly intelligent, because it lacks the degree of self-awareness it should have if it were".

Stop conflating certain cognitive capabilities with "true intelligence". If you want to know whether LLMs can do a certain cognitive task, devise a set of fairly easy tests they couldn't pass without said capability. Set aside your preconceptions about intelligence and just look at how it performs on those tests.

The error in this thought process is assuming self learning is conflated with self awareness. I'm making no such assertion. My simple assertion is that LLM's does not operate distinctively different than other machine learning models when it comes to data training. They still need to be feed the data much the same way as other model types and access that data in the same way. When models can do this without it's human overloads, self aware or not, I will believe we've moved on from machine learning into something closer to" true AI"

 

[bit_user]

This and other statements get to the bottom of our disagreement. I'm classifying things by their essence while you want to classify them by their utility.

My original claim was about the ability of generative AI to perform specific cognitive tasks. That's all I'm saying. I think it's not productive to get side-tracked on a discussion of whether AI is "truly intelligent", which seems to be where you and @JamesJones44 are coming from.

 

[bit_user]

The error in this thought process is assuming self learning is conflated with self awareness. I'm making no such assertion. My simple assertion is that LLM's does not operate distinctively different than other machine learning models when it comes to data training. They still need to be feed the data much the same way as other model types and access that data in the same way. When models can do this without it's human overloads, self aware or not, I will believe we've moved on from machine learning into something closer to" true AI"

Whether or not they do online learning is entirely separate from the question of whether they can perform certain types of reasoning.

 

[JamesJones44]

Whether or not they do online learning is entirely separate from the question of whether they can perform certain types of reasoning.

I'm not arguing the reasoning part or even the imitation of it. I'm arguing that LLM is still an evolution of machine learning, in that it still relies on inference for all decisions and it doesn't have cognitive abilities (though it does a good job of imitating them) thus making closer to machine learning than it is a new category of AI or "true" artificial intelligence.

NOTE: When I say cognitive abilities I don't mean sentience, self aware or alive. It does imitate the pattern recognition part of cognitive abilities but it's not truly cognitive.

 

[bit_user]

I'm arguing that LLM is still an evolution of machine learning, in that it still relies on inference for all decisions and it doesn't have cognitive abilities

I think you're making an artificial distinction. Either that, or maybe you're being mislead by usage of the term "inference".

 

[Deleted member 2962262]

A new ChatGPT feature lets you upload files full of data or code, but it also makes them vulnerable to exfiltration.

ChatGPT's New Code Interpreter Has Giant Security Hole, Allows Hackers to Steal Your Data : Read more

But judging by the unfolding cases in the courts, the very development of these LLM bots is based on industrial scale theft of intellectual property by the developers themselves.

 

[JamesJones44]

I think you're making an artificial distinction. Either that, or maybe you're being mislead by usage of the term "inference".

We'll have to be settle on agreeing to disagree on whether LLM is machine learning or a new class of AI

 

[bit_user]

We'll have to be settle on agreeing to disagree on whether LLM is machine learning or a new class of AI

The main problem with debating whether something is really "AI" is the continual goalpost movement by non-experts in the field. All of the long-time AI academics and practitioners seem to agree that generative AI fits the definition - it's only the general public that seems to object.

The word "artificial" is there to indicate that it shouldn't be seen as truly intelligent, but merely something capable of intelligent behavior. Yet, the popular sentiment seems to be pretending as if the qualifier isn't there and holding AI up to the benchmark of a human-level intelligence, before we're willing to accept that it's actually behaving in intelligent ways.

I think it's informative to compare with how we talk about animals long regarded to be intelligent. Take crows, for instance. They can perform numerous cognitive tasks not widely seen among members of the animal kingdom and are generally regarded as comparatively intelligent animals. Such claims are not regarded as controversial - people are willing to accept that they have some advanced cognitive skills and are willing to characterize it as intelligence, without requiring them to demonstrate the full range and magnitude of human cognition.

I think the way to move forward is simply to define a set of cognitive capabilities and measure how well leading AI models perform them. On that note, here's a recent paper that set forth a benchmark for measuring how well AI performs logical reasoning:

Dataset	LogiQA 2.0 test	LogiQA 2.0 zh test	ReClor dev	AR-LSAT test	LogiQA 2.0 ood
Size	1572​	1594​	500​	230​	1354​
Human avg.	86​	88​	63​	56​	83​
human ceiling	95​	96​	100​	91​	99​
RoBERTa	48.76​	35.64​	55.01​	23.14​	33.22​
ChatGPT (API)	52.37​	53.18​	57.38​	20.42​	38.44​
GPT-4 (Chat UI)	75.26 (73/97)​	51.76 (44/85)​	92 (92/100)​	18.27 (19/104)​	48.21(54/112)​
GPT-4 (API)	72.25​	70.56​	87.2​	33.48​	58.49​

Table 1: ChatGPT and GPT-4 performance on the Logical multi-choice machine reading comprehension task
(accuracy %). “LogiQA 2.0 zh test” refers to the test set of the LogiQA 2.0 Chinese version. “LogiQA 2.0 ood”
represents the out-of-distribution data of LogiQA 2.0.

Source: Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4
​

There are more test results in the paper, but it's too much trouble to copy & paste them here.

From the abstract:

... Experiment results show that ChatGPT performs significantly better than the RoBERTa fine-tuning method on most logical reasoning benchmarks. ... The results show GPT-4 yields even higher performance on most logical reasoning datasets. Among benchmarks, ChatGPT and GPT-4 do relatively well on well-known datasets like LogiQA and ReClor. However, the performance drops significantly when handling newly released and out-of-distribution datasets. Logical reasoning remains challenging for ChatGPT and GPT-4, especially on out-of-distribution and natural language inference datasets.
​

Note that they don't deny that these models are capable of reasoning - rather concluding simply that they're not yet very good at it.

 

[JamesJones44]

It will impact many more jobs, and many more industries

While no one can predict the pace things will move, when looking out 10 years, I'm not overly worried about this aspect of LLM/current AI technology and the societal impact. It will have impact, that's given, but I'm not sure how many jobs it will truly "replace" within say a 10 year span. If anything in that time I think the job gains due to AI/ML vs job loses due to AI/ML will roughly net out. The question is will it cause pay reductions and what happens as it evolves, that part is harder to say. 20 years from now might be a different story.

The reason I feel this way is the pace of improvement over the 8 years I've been aware and worked with ML has actually been way slower than I thought it would be. For example, I thought self driving cars would have been well out of fixed areas and "beta" by now, yet we are still looking like we are 5 or more years away from cars being able to truly drive themselves simply using a GPS and mapping with AI/ML (Waymo for example uses a combination of AI/preprogrammed routes, Tesla is trying, but it's still has a far number of issues). Even when it comes to computer vision, we still have human analysis pouring over data to verify its authenticity and update models, 8 years ago I would have thought that would have been a thing of the past by now. LLM was a big jump, no doubt about it, but classifiers, deep learning, neural networks, etc. were all also big jumps. LLM is just in peoples faces and it is the first one that average user can interact with so it gets a lot of attention, but the pace is still slower than I would expect.