News and discussion about NLP

News and discussion about NLP

by Giorgio Satta -
Number of replies: 16

I am opening this new thread where people can post  news and ideas regarding the NLP world. You are very welcome to share links and discussions.

Given the constant acceleration of the field, I expect that this thread will be quickly populated with many posts!

In reply to Giorgio Satta

Re: News and discussion about NLP

by Giorgio Satta -

To start with, Antropic has just announced the new Claude 3 model family. You can read the announcement and a short introduction here.

In reply to Giorgio Satta

Re: News and discussion about NLP

by VINAYAK KUMAR -
I found this recent paper FuseChat: Knowledge Fusion of Chat Models (available at https://arxiv.org/abs/2402.16107) interesting, moving beyond the traditional approach of training a LLM from scratch, an alternative option is to combine existing LLMs into a new, more powerful one.
Wondering if we could club Claude3,Chatgpt and Gemini ?
In reply to Giorgio Satta

Re: News and discussion about NLP

by VINAYAK KUMAR -
I found about recent paper on Echo embedding(can be found at https://arxiv.org/abs/2402.15449v1) which is similar to our lecture 4 but one week older news than Claude 3 model family news. The paper is on how echo embeddings improve over classical embeddings(which is word2Vec,etc).
In reply to Giorgio Satta

Re: News and discussion about NLP

by Giorgio Satta -

Pika, a very recent generative AI model for text2video, has now incorporated sound in its output. This works either through your prompt, or else it is automatically generated based on the content of your video.

You can see their demo here.

In reply to Giorgio Satta

Re: News and discussion about NLP

by Giorgio Satta -

Interesting post on X/Twitter by Yann LeCun here on the comparison of language and vision wrt information bandwidth.

Errata: 16 million should be 1.6 million

In reply to Giorgio Satta

Re: News and discussion about NLP

by VINAYAK KUMAR -
I glanced through Novel Snow crash(Which has computational linguistics theme) and "Neurolinguistic hacking" could be an idea with real world applications like healthcare and therapy, Human computer interaction and neurocognitive research.
In reply to Giorgio Satta

Re: News and discussion about NLP

by Giorgio Satta -

To which extent can we say that LLMs understand language? Here are two quite different views, in answer to the question.

The first article summarizes the results in a recent paper by the authors on meaning representations, and presents a positive view of the relation between meaning and LLMs. You can find this blog-post at this link

The second article has a more critical perspective on this problem. It introduces the so-called missing text phenomenon, that characterizes the communication process in natural language. You can find this blog-post at this link

In reply to Giorgio Satta

Re: News and discussion about NLP

by VINAYAK KUMAR -
Inspired by Novel "The Diamond age" where Nell learns martial arts quickly from the highly interactive book which adapts based on verbal inputs from Nell. If we integrate this idea with Matrix Movie 1 minute Kung Fu learning, Neuralink brain chips and few muscle tension adjusting nanobots.
Here is how I think It could be achieve using NLP:
A)Linguistic Compression-Through advanced NLP, complex martial arts maneuvers are distilled into their essence, conveyed in a language format that the brain can rapidly decode. This process utilizes the brain's ability to understand and process language efficiently, engaging areas like the Broca’s area (involved in language processing and speech production) and Wernicke’s area (involved in understanding spoken and written language).
B)Spaced Retrieval Practice- idea would be to use rapid, successive, micro-retrievals to simulate the effect of spaced learning, artificially creating the impression of elapsed time by manipulating neural activation patterns. This could hypothetically enhance synaptic plasticity in a compressed timeframe, a process normally spread out over longer periods.
C)Rapid Consolidation Techniques-Rapid consolidation would rely on NLP-generated content to trigger emotional and sensory engagement, simulating the vivid, immersive experiences that lead to stronger memory encoding. This could involve creating highly detailed, sensory-rich narratives around martial arts movements, engaging the amygdala (which plays a significant role in emotion and memory) and the sensory cortices.
By leveraging the brain’s encoding of emotional and sensory experiences, this approach seeks to accelerate the consolidation of motor skills into long-term memory, effectively bypassing the slow, repetitive physical practice.

Very unsure, But just an idea.
In reply to Giorgio Satta

Re: News and discussion about NLP

by Giorgio Satta -

For Italian students only:

Next week at DEI, on April 12th 12:30 classroom Ke, there will be an interesting (Italian only) presentation on LLMs, titled 'Modelli generativi del linguaggio: capacità superumane o pappagalli statistici?' See leaflet in attachment, just in case you are interested.

Attachment locandina_2024.jpg
In reply to Giorgio Satta

Re: News and discussion about NLP

by Giorgio Satta -

On 18 April, Meta released Llama 3, trained on a 15 trillion tokens dataset and with wonderful performance on popular benchmark MMLU (Massive Multitask Language Understanding). It's the latest AI model to be offered by Meta free of charge and with a relatively open (though not open-source) license that lets developers deploy it in most commercial apps and services.

The official paper on Llama 3 is not yet out, but you can read more about the model at this link.

In reply to Giorgio Satta

Re: News and discussion about NLP

by Giorgio Satta -

I am reading with interest the Artificial Intelligence Index Report 2024, issued by the Human Centered Artificial Intelligence group, Stanford Univ., available at this link.

It's a pretty long document, but to get an idea of most important points you can quickly look at pages 14--26, where you can find a highlight of the 10 chapters of the report.

We might talk some about this more at the end of the nlp course, in the wrap-up lecture.

In reply to Giorgio Satta

Re: News and discussion about NLP

by Giorgio Satta -

I don't want to distract you from the ongoing final exam session, but I think that the survey paper at this link might be of interest to some of you. It is a very long document, but you can just look into the introduction and Figure 1, which I have found very interesting.

In reply to Giorgio Satta

Re: News and discussion about NLP

by Alberto Formaggio -
One thing I believe might be worrying is that everyone now is following this LLM hype blindly. It's like the solution to any problem can be done by LLMs and RAG. RAG doesn't make sense in a given context? Let's throw it in anyways, adjust a little bit the prompt adding some words to make the model reason better and here we have a new paper saying how amazing LLMs are.
The same (or better) could have been achieved with some SOTA models but instead of improving the SOTA, people just keep going with LLMs.

I think in a few years we'll see if this LLM hype actually led to superhuman models or to a dead end. In the latter case, we may have lost some years in improving the SOTA for (hopefully not) nothing.