This loop continues leaving you frustrated
If you've used LLMs a lot, you know this feeling!
Enter UI components embedded directly in the chat
Grokbot solves this by putting a basic email editor in chat itself
Small thing but fixes a real limitation of chat and iterations 4/n
1
34
> opened codex web
> codex asks to integrate with slack
> me: why not? i can delegate all things directly from slack
> started a task from a slack thread
> codex returns the task link
> coworker opens the task link without any auth and is able to see everything
> tried the link in the incognito mode
> link still opens up with all of the code/details for anyone out there to see
> tried for 30 min to find a setting to disable this.
> found nothing
> manually disable links so far one by one
> disables codex in slack :(
> back to claude/codex duo in tmux
1
11
1,431
Excited to announce our Grounded Language Model (GLM) is now available through our API! 🚀
GLM is the best in the world at grounding in your context and achieves state-of-the-art results on the FACTS grounding benchmark🥇. This means:
- Less hallucinations: GLM grounds fully in the context and knowledge you provide, only leveraging its parametric knowledge when necessary. 🎯
- More trustworthy answers: By focusing solely on context relevant to the query from your provided knowledge and filtering out irrelevant information, GLM delivers more accurate and reliable responses. ✅
- Enterprise-grade: Beyond accuracy, high-stake production deployments require better handling of inaccuracies. GLM gracefully declines to answer when it can't find relevant context in your provided knowledge. 🛡️
1
3
333
Benchmark here - bird-bench.github.io/
Read more about structured data and other capabilities available in the platform here - contextual.ai/blog/unstructu….
We also have extensions to standard structured databases and data warehouses like BigQuery, Snowflake, and Redshift.
2
278
We achieved the #1 ranking🎉with 75.63% test set accuracy on the BIRD-SQL benchmark, which evaluates text-to-SQL capabilities in a variety of complex domains. 🏆🚀
Available on @contextualai platform. Work led by the amazing @sheshanshag!
2
4
25
1,398
YouTube: piped.video/TQKDU2mhKEI
Spotify: creators.spotify.com/pod/sho…
Podcast Recap: connorshorten300.medium.com/…
4
350
Proud to announce that the Contextual AI Platform is now generally available!
We set new state of the art across the RAG stack, while consistently outperforming GPT-4o and Claude-3.5-Sonnet-powered RAG agents 🥇.
This sota performance extends across every critical dimension of building a RAG agent - from document understanding and retrieval to structured data understanding and grounded generation - all delivered with enterprise-grade security, scalability, and available through our platform API.
Best of all, through advanced specialization capabilities, our platform enables your Contextual RAG agents to grow and adapt with your enterprise, achieving even greater performance gains by tailoring to your specific use cases.
Links in the 🧵👇.
5
11
125
25,061
Just a passing thought as I was reading the Mixtral blog post possibly a coincidence: the languages supported by Mixtral and those in Red Pajama v2 are exactly the same. Mixtral is likely trained on a clean subset of RedPajama v2. 🤔
2
6
1,283
IDEFICS, the first open source GPT4-style model is now available.
You can do image+text conversations with IDEFICS like a visual ChatGPT. Work done with my awesome former colleagues at @huggingface . 🧵(1/n)
3
6
22
2,983
Contextual AI is now in the GenAI 50 list moving up from the GenAI 100 list previously from @CBinsights! 🚀
Meet the GenAI 50 — our first-ever list of the most promising private generative AI companies across the globe. These winners are working on foundation models, code generation, AI assistants, and more.
cbi.team/448tvny
2
22
2,244
Found this little nugget in Accelerate today which makes writing distributed training/inference code decoupled from the data source very easy.
You can achieve this in a few lines of code now.
This previously required a lot of effort and proper distributed setup.
huggingface.co/docs/accelera…
1
4
21
2,863
When we created SuperGLUE, we didn't expect it to saturate within 6 months!
Our new post shows how the progress has accelerated since then and how the benchmarks are saturating even faster. Inspired from our teaser in Dynabench paper: contextual.ai/plotting-progr…
1
5
38
5,395
Reading party hosted by @bhutanisanyam1 on LENS: piped.video/watch?v=88uHdoij….
First of a kind?
Making LLMs multi-modal without training! 🙏
An amazing read on how to effectively combine captioning models with LLMs to outperform multi-modal models.
The main challenge w multi-modal LLMs is the much larger compute requirement or need for larger datasets.
This paper suggests an effective approach of avoiding this:
- Image Tags: Use CLIP to generate Tags of an image
- Image Attributes: Again, CLIP to detect objects in images
- Intensive Captioning: BLIP for generating captions of an image
All of these are then fed to a LLM, in this case FLAN-T5 which then acts as a multi-modal LLM.
The authors show that this approach easily outperforms specially trained image LLMs.
Personally, I’ve played a lot with GPT-4 and I’m sure it can perform even stronger than the above approach.
If you’re interested, I’ll be hosting a paper reading livestream on this today:
piped.video/watch?v=88uHdoij…
2
8
1,970
In LENS 🔎, we leverage advances in multimodal field to describe an image using combination of modules. This description is then passed to an LLM with the corresponding query.
Surprisingly, this simple baseline can easily beat more sophisticated models like Flamingo by 9% [4/N]
1
2
2
667
LLMs can now handle computer vision and visual reasoning tasks. 🔥
Introducing LENS 🔎, a modular approach that empowers any LLM to perceive and reason about images. 👁️
Demo: lens.contextual.ai
Blog: contextual.ai/introducing-le…
A 🧵 on why we did it [1/N]
Announcing LENS 🔎, a framework for vision-augmented language models.
- Outperforms Flamingo by 9% (56->65%) on VQAv2
- Eliminates the additional cost of multimodal pre-training
Demo: lens.contextual.ai
Blog+Paper+Code: contextual.ai/introducing-le…
A 🧵 [1/N]
3
33
152
34,890
Thrilled to see @ContextualAI making waves in the "Foundation Models" section of @CBinsights' top 100 most promising AI companies. 🎉
Boom: The most promising AI companies around the world.
This year's winners include startups working on generative AI infrastructure, emotion analytics, general-purpose humanoids, and more. #AI100 cbi.team/3PkNdIM
3
1
35
6,465
For our super stealthy startup (more details soon), we're benchmarking cloud compute clusters.
Hoping for some wisdom of the crowds: What should we look out for? What should we measure? 💸
8
3
26
9,747
Compared to prior work and CLIP, FLAVA again performs very competitively on vision-and-language tasks as well. (9/N)
1
6
Compared to BERT and RoBERTa, FLAVA does reasonably well when prompted using simple manual prompts. We use manual prompts from openreview.net/pdf?id=YLglAn… directly without any specific fine-tuning. (8/N)
1
1
5
Compared to CLIP, FLAVA does well on image classification tasks with a linear probe even though it is pretrained on significantly less data (400M vs 70M). FLAVA outperforms CLIP significantly on text understanding tasks showing its superior performance across modalities. (7/N)
1
1
3
FLAVA leverages unimodal data which is abundantly available along with paired image-text data to work on image and text understanding tasks as well as vision-and-language tasks. We achieve this by pretraining on unimodal, cross-modal and multimodal pretraining tasks. (4/N)
1
1
6
SotA V&L models often target specific modalities or tasks. With our fully open source FLAVA model, we aim to move beyond that. FLAVA works on a wide range of 35 tasks like image classification, NLU, retrieval and visual reasoning. Presenting it tomorrow at #CVPR2022. A 🧵 (1/N)
1
18
88
We just pushed ImageNet behind gated access for compliance logistics.
Make sure to login first using huggingface-cli since the dataset is gated, click for access at huggingface.co/datasets/imag… and then call `load_dataset("imagenet-1k", use_auth_token=True)` to access the dataset.
4
Now, ImageNet-1K can be downloaded in just two lines using the @huggingface datasets library!
Thanks to @MarioSasko, manual download is no longer needed.
ImageNet-1k dataset is now available on @huggingface 🤗 hub. Try it now at: huggingface.co/datasets/imag….
We are continually improving the computer vision and multimodal support in 🤗 ecosystem. Look forward to more exciting things coming out soon.
2
17
96
Can SotA multimodal models "ground" caption properly?
In WinoGround, for an image, models need to differentiate 2 captions which have exact same words but are differently ordered and visually depict very different things
Even CLIP/FLAVA don't beat a random baseline! #CVPR2022
Happy to announce our new CVPR paper - Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality.
All tested SOTA multimodal models perform very poorly on our new vision-language eval dataset.
Paper: arxiv.org/abs/2204.03162
#CVPR2022, #NLProc
1/5
1
1
7
Wanna try fooling the model for yourself? Try your best on Dynabench which we used to collect our examples:
dynabench.org/tasks/12/creat…
Dataset and evaluation server coming in the near future!
5/6
1
3
The adversarial examples we collected are not as unnatural as you might think, and many showcase VQA models' known weaknesses, such as counting and dealing with text. Our findings clearly show that this task is far from solved, and that there is a lot of room for improvement.
3/6
1
4
Current VQA datasets have started plateauing and are approaching saturation w.r.t. human performance. In this work, we asked human adversaries to find examples that MoViE+MCAN, the 2020 VQA challenge winner, gets wrong. SotA V&L models do very poorly on our new AdVQA dataset.
2/6
1
4
Vision and language research has made great progress, but how good are we really, and what are we still missing? We examine these questions in Human-Adversarial VQA:
arxiv.org/abs/2106.02280
1/6
2
24
114
ERNIE from Baidu becomes first submission to cross 90 on GLUE Benchmark.
Original ERNIE paper: arxiv.org/abs/1907.12412v1
2
6
45
🚨 GLUE SoTA Alert 🚨
ALBERT (Ensemble) by Google Language team achieves 89.4 score on the GLUE benchmark. 😱
Looking forward to the manuscript.
6
35
147
Checkout and discuss our @cvpr2019 poster #196 "Towards VQA Models That Can Read" at 3:20PM. Learn how current VQA models fail to answer questions about image text and our steps toward solving it. More details + TextVQA dataset + Starter Code available at textvqa.org.
8
15
.@agrimgupta92 presenting his work on Large Vocabulary Instance Segmentation (LVIS) at poster no. 82 #CVPR2019
2
2
17
If you at #NeurIPS2018, come checkout our talk and poster tomorrow on "Learning when to communicate in multiagent scenarios" in Emergent Communication Workshop #EMECOM2018.
1
2
18
