How Accurate Are Multimodal AI Models for Pickleball Video Analysis? A Benchmark Study of Gemini and Qwen

By: Smruthi Karthik

In our experiment with Gemini Free Version, Gemini (Google AI Pro), and Qwen 3.6 Plus, none of the models demonstrated the level of accuracy and reliability required for professional pickleball analysis.

Among the tested models, Gemini Free Version performed the best in terms of structural accuracy, correctly identifying the number of rallies and timestamps for each rally, but it still made notable errors in scoring, shot classification and overall event understanding.

Qwen 3.6 Plus often failed to correctly identify the rally winner but managed to identify the point winner, probably leveraging cues from scoreboard, though its overall rally-level analysis remained inconsistent.

Introduction

A demonstration by Google on YouTube, in late 2025, suggested that its Gemini model can analyze pickleball match footage and provide suggestions on how to improve a player’s movements. While this capability appears promising, independent validation under realistic conditions is necessary for these promising claims.

This led us to design an experiment to test how well current multimodal AI systems actually perform when tasked with analyzing a real pickleball match. Instead of relying on a single model, we compared it against other commercially available systems to understand their relative strengths and limitations.

The experiment aimed to identify which model delivers the most reliable results in tracking points, classifying shots, and segmenting rallies from video input using a consistent prompt, without any sport-specific training.

Testing Conditions

1. All experiments were conducted on April 16, 2026.
2. The models used were Gemini 3 Flash (Google AI Pro) and Qwen 3.6-Plus.
3. Each model was tested using the same prompt to maintain consistency and ensure a fair comparison.
4. The input consisted of a broadcast-quality pickleball video.
5. The clip included continuous gameplay between two players, allowing for complete rally and point analysis.
6. The total duration of the video was 120 seconds.(Sample clip is given below)

Pickleball video analysis prompt

Task:

Analyze the given pickleball match video and extract detailed match statistics. Follow all instructions carefully and ensure maximum accuracy.
Scoring System (Important)

• Side-out scoring: Only the serving player/team can win a point
• Point Calculation: A point is awarded only when the serving player wins the rally
• If the receiving player wins the rally, no point is awarded (serve changes)

Player Identification

• Anna Leigh Waters: Wearing white t-shirt, positioned closer to the camera, right-handed
• Chao Yi Wang: Wearing red t-shirt, positioned farther from the camera, right-handed

Required Outputs

1. Points Won

• Total points won by Anna Leigh Waters (white shirt)
• Total points won by Chao Yi Wang (red shirt)

2. Point-by-Point Description

For each rally/point, provide:

• Who won the rally
• A short, simple description of how the point was won

Examples:

• “Anna Leigh hits a cross-court forehand winner”
• “Chao Yi wins after a long rally when Anna misses into the net”
• “Anna wins the point with a strong smash at the net”

Strict rule:

• Do not hallucinate or assume details
• Only describe what is clearly visible in the video
• If unclear, write: “unclear from video”

3. Rally Definition (Critical for Timestamps)

• A rally starts at the exact moment when:
  • The serve is made, and
  • The paddle makes contact with the ball
• A rally ends when:
  • The point outcome is decided and the ball is no longer in play
• Only count active play time, ignore gaps between rallies

4. Shot Classification Rules

• Forehand Shot: Right-handed player hits on dominant (right) side, palm facing forward
• Backhand Shot: Right-handed player hits on non-dominant (left) side, back of hand leading

5. Shot Counting (Screen-Based Orientation)

Anna Leigh Waters (near camera):

• Forehand = right side of screen
• Backhand = left side of screen

Count:

• Total forehands
• Total backhands

Chao Yi Wang (far from camera):

• Forehand = left side of screen
• Backhand = right side of screen

Count:

• Total forehands
• Total backhands

6. Rally Analysis

For each rally:

• Start timestamp (mm:ss) → at serve contact
• End timestamp (mm:ss) → when rally ends
• Total shots in rally (each paddle-ball contact = 1 shot)
• Clearly indicate whether a point was awarded or not based on the scoring system

Output Format (Strictly Follow)

A. Match Summary

• Anna Leigh Waters Points: X
• Chao Yi Wang Points: X

B. Shot Breakdown

Anna Leigh Waters:

• Forehands: X
• Backhands: X

Chao Yi Wang:

• Forehands: X
• Backhands: X

C. Rally + Description Table

Rally No.	Start Time	End Time	Shots	Rally Winner	Point Awarded (Yes/No)	Description
1	00:12	00:25	6	Anna Leigh	Yes	Wins with a forehand shot after a short rally
2	…	…	…	…	…	…

Critical Accuracy Rules

• Do not guess or hallucinate
• Only include visually confirmed actions
• If unsure, write “unclear from video”
• Use frame-by-frame analysis if needed
• Ensure totals are consistent
• Ensure points follow the side-out scoring system strictly

Ground Truth from the Pickleball Video clip

A. Match Summary

• Anna Leigh Waters Points: 1
• Chao Yi Wang Points: 2

B. Shot Breakdown

Anna Leigh Waters:

• Forehands: 9
• Backhands: 10

Chao Yi Wang:

• Forehands: 9
• Backhands: 10

C. Rally + Description Table

Rally No.	Start Time	End Time	Shots	Rally Winner	Point Awarded	Description
1	00:02	00:08	4	Chao Yi Wang	No	Chao hits a backhand cross-court and Anna struggles to get behind the ball to return it.
2	00:21	00:25	4	Chao Yi Wang	Yes	Anna’s shot goes wide right outside the court, giving Chao the point.
3	00:34	00:39	5	Anna Leigh Waters	No	Chao tries to hit a forehand cross-court shot but hits the ball into the net.
4	00:51	01:06	11	Anna Leigh Waters	Yes	Anna hits a backhand down the line to win the rally and point.
5	01:20	01:29	7	Chao Yi Wang	No	Anna’s forehand goes wide right, giving Chao the rally.
6	01:43	01:51	7	Chao Yi Wang	Yes	Chao hits a backhand down the line to win the rally.

Results Generated by Multimodal AI Models

Summary

Gemini (Free Version)

Output

This analysis of the pickleball match between Anna Leigh Waters (white shirt) and Chao Yi Wang (red shirt) follows side-out scoring rules and strictly observes the provided footage.

A. Match Summary

• Anna Leigh Waters Points: 2
• Chao Yi Wang Points: 1

B. Shot Breakdown

Anna Leigh Waters

• Forehands: 11
• Backhands: 18

Chao Yi Wang

• Forehands: 7
• Backhands: 19

C. Rally + Description Table

Rally No.	Start Time	End Time	Shots	Rally Winner	Point Awarded	Description
1	00:02	00:08	6	Anna Leigh	Yes	Anna Leigh serves and wins the point after Chao Yi hits a backhand into the net.
2	00:20	00:25	5	Chao Yi	No	Anna Leigh serves, but Chao Yi wins the rally when Anna Leigh hits a forehand into the net (Side-out).
3	00:33	00:40	6	Chao Yi	Yes	Chao Yi serves and wins the point after Anna Leigh returns a backhand into the net.
4	00:52	01:07	17	Anna Leigh	Yes	Anna Leigh serves; after a long baseline rally, she hits a backhand passing winner down the line.
5	01:20	01:29	9	Chao Yi	No	Anna Leigh serves, but Chao Yi wins the rally when Anna Leigh’s backhand fails to clear the net (Side-out).
6	01:42	01:52	12	Anna Leigh	No	Chao Yi serves; Anna Leigh wins the rally with a powerful forehand winner after moving into the court (Side-out).

Technical Notes

• Scoring: Under side-out rules, Anna Leigh earned points in Rally 1 and Rally 4. Chao Yi earned a point in Rally 3.
• Orientation: As per instructions, shots for Anna Leigh (near camera) were classified as Backhands on the left of the screen and Forehands on the right. For Chao Yi (far camera), Forehands were on the left and Backhands on the right.
• Rally 4 was the longest sequence, featuring 17 distinct paddle-ball contacts.

Evaluation Parameters for Gemini Free Version

• Speed / Time Taken
  The thinking mode of Gemini Free Version took approximately 56 seconds to analyze the pickleball video and generate a response.

• Timestamp Accuracy
  All the timestamps were correct, accurately matching the exact moments of each rally.

• Event Accuracy
  The model correctly identified the rally winner in 3 out of 6 rallies. However, most of the descriptions were inaccurate. Only in the 4th rally was the event described correctly – and the rally winner, point winner, and description were accurate, although the number of shots in that rally was incorrect.

• Summary

Overall, there are clear discrepancies in the results generated by Gemini Free Version. The match summary was incorrect, with the actual score (Anna Leigh Waters: (1), Chao Yi Wang: (2) being reversed by the model, showing a misunderstanding of scoring. Additionally, the shot breakdown was overestimated for both players.

Gemini(Google AI Pro) Results

Output

Following the provided side-out scoring rules and frame-by-frame analysis of the video, here is the detailed match breakdown.

A. Match Summary

• Anna Leigh Waters Points: 4
• Chao Yi Wang Points: 1

B. Shot Breakdown

Anna Leigh Waters:

• Forehands: 14
• Backhands: 15

Chao Yi Wang:

• Forehands: 9
• Backhands: 18

C. Rally + Description Table

Rally No.	Start Time	End Time	Shots	Rally Winner	Point Awarded	Description
1	00:04	00:08	5	Anna Leigh Waters	Yes	Anna serves and wins the point with a powerful cross-court forehand winner.
2	00:20	00:25	5	Chao Yi Wang	No	Anna serves, but Chao Yi wins the rally when Anna misses a backhand into the net.
3	00:33	00:39	7	Anna Leigh	No	Chao Yi serves, but loses the rally after a baseline exchange when her backhand goes long.
4	00:52	00:57	6	Anna Leigh	Yes	Anna serves and wins the point after Chao Yi hits a backhand into the net.
5	00:59	01:06	15	Anna Leigh	Yes	Anna wins a long baseline rally with a sharp forehand winner down the line.
6	01:21	01:29	10	Anna Leigh	Yes	Anna serves and maintains control until Chao Yi misses a forehand wide.
7	01:43	01:50	8	Chao Yi Wang	Yes	Chao Yi serves and wins her first point after Anna hits a backhand into the net.

Evaluation Parameters

• Speed / Time Taken
  The thinking mode of Gemini Google AI Pro took approximately 45 seconds to analyze the video and generate a response.

• Timestamp Accuracy
  The model recorded 7 rallies instead of 6 and got 3 of the timestamps incorrect.

• Event Accuracy
  The model got all the descriptions wrong and failed to correctly identify who scored the points. The output contained multiple hallucinations and the overall score was completely incorrect. While it correctly identified Chao Yi Wang’s number of forehands, as well as the rally winner in 4 instances and the point winner twice, it failed to accurately capture backhands and the opponent’s shot data.

• Summary
  Overall, the results generated by Gemini Google AI Pro were inconsistent. Despite slightly faster processing speed, the model struggled with core aspects such as scoring, event identification, and shot tracking. Frequent hallucinations, incorrect timestamps, and inaccurate descriptions significantly reduced the reliability of its analysis.

Qwen 3.6 Plus

Output

A. Match Summary

Anna Leigh Waters Points: 1
Chao Yi Wang Points: 2

B. Shot Breakdown

Anna Leigh Waters:
Forehands: 21
Backhands: 22
Chao Yi Wang:
Forehands: 21
Backhands: 20

C. Rally + Description Table

Rally No.	Start Time	End Time	Shots	Rally Winner	Point Awarded (Yes/No)	Description
1	00:02	00:08	11	Chao Yi Wang	No	Anna hits a backhand into the net after a short rally.
2	00:14	00:25	16	Chao Yi Wang	Yes	Anna hits a forehand long/out of bounds.
3	00:33	00:40	10	Chao Yi Wang	No	Anna hits a backhand into the net (Side-out).
4	00:50	00:58	9	Chao Yi Wang	No	Anna hits a forehand into the net.
5	01:00	01:06	7	Anna Leigh Waters	No	Chao hits a backhand long/out of bounds (Side-out).
6	01:17	01:30	10	Anna Leigh Waters	Yes	Chao hits a backhand long/out of bounds.
7	01:34	01:51	22	Chao Yi Wang	Yes	Anna hits a backhand into the net after a long rally.

Evaluation Parameters of Qwen 3.6 Plus

1. Speed / Time Taken
   The thinking mode of Qwen 3.6 Plus took approximately 4 minutes and 40 seconds to analyze the video and generate a response.
2. Timestamp Accuracy
   The model recorded 7 rallies instead of 6 and got only 3 of the timestamps correct.
3. Event Accuracy
   Qwen 3.6 plus reported the correct overall match score (Anna Leigh Waters: 1, Chao Yi Wang: 2) but the rally-level analysis was inconsistent. It showed major errors in counting the number of shots, overestimating forehands and backhands for both players. Rally winners and point outcomes were frequently incorrect, and most descriptions did not match the actual events in the video.
4. Summary
   Overall, Qwen 3.6 Plus demonstrated poor reliability in detailed match analysis. Despite taking the longest processing time, it struggled with rally identification, shot accuracy and event interpretation. The combination of incorrect rally counts, inaccurate descriptions and inflated shot data significantly reduced the credibility of its output.

Best Performing Model

Among the models tested, Gemini Free Version delivered the strongest overall performance. It correctly identified the total number of rallies and achieved perfect timestamp accuracy, with all rally timings matching the ground truth. While it showed moderate event accuracy by correctly identifying the rally winner in 3 out of 6 rallies and fully capturing only one rally correctly it still outperformed the other models, which had greater issues with hallucinations, incorrect rally counts, and poor timestamp alignment. However, Gemini Free Version still had notable limitations, which makes it unfit for reliable analysis of professional pickleball.

Conclusion

The results of this experiment suggest that current commercially available multimodal AI models are not yet capable of delivering the level of accuracy required for reliable pickleball match analysis, even when provided with structured prompts. While prompt engineering can improve outputs to some extent, the gap between model predictions and ground truth indicates that deeper improvements are needed. A more effective approach would involve fine-tuning models on sport-specific datasets. 

SPAN by Banyan Board is a video labeling platform designed to help teams efficiently annotate events within video footage. It allows large groups of annotators to work collaboratively, making it easier to create high-quality datasets that can be used to fine-tune multimodal AI models for sports performance analysis.

To learn more, visit our website and explore our knowledge base.