Recommended
More Related Content
Viewers also liked
Viewers also liked (16)
Similar to MediaEval 2016: A Multimodal System for the Verifying Multimedia Use Task
Similar to MediaEval 2016: A Multimodal System for the Verifying Multimedia Use Task (20)
More from multimediaeval
More from multimediaeval (20)
Recently uploaded
Recently uploaded (20)
MediaEval 2016: A Multimodal System for the Verifying Multimedia Use Task
- 1. Linkmedia team participation: a multimodal system for the Verifying Multimedia Use task C´edric Maigrot, Vincent Claveau, Ewa Kijak, Ronan Sicre October 20, 2016 Maigrot, Claveau, Kijak, Sicre Linkmedia team participation: a multimodal system for the Verifying Multimedia Use taskOctober 20, 2016 1 / 3
- 2. A multimodal system for the Verifying Multimedia Use task Text-based Is the message style-wise similar to known hoax? Maigrot, Claveau, Kijak, Sicre Linkmedia team participation: a multimodal system for the Verifying Multimedia Use taskOctober 20, 2016 2 / 3
- 3. A multimodal system for the Verifying Multimedia Use task Text-based Is the message style-wise similar to known hoax? Source-based Presence of a trustworthy source in the text content? Maigrot, Claveau, Kijak, Sicre Linkmedia team participation: a multimodal system for the Verifying Multimedia Use taskOctober 20, 2016 2 / 3
- 4. A multimodal system for the Verifying Multimedia Use task Text-based Is the message style-wise similar to known hoax? Source-based Presence of a trustworthy source in the text content? Image-based The image is already known? Maigrot, Claveau, Kijak, Sicre Linkmedia team participation: a multimodal system for the Verifying Multimedia Use taskOctober 20, 2016 2 / 3
- 5. A multimodal system for the Verifying Multimedia Use task Text-based Is the message style-wise similar to known hoax? Source-based Presence of a trustworthy source in the text content? ↓ Image-based The image is already known? → Combinaison approach Can the three previous predictions help? Maigrot, Claveau, Kijak, Sicre Linkmedia team participation: a multimodal system for the Verifying Multimedia Use taskOctober 20, 2016 2 / 3
- 6. A multimodal system for the Verifying Multimedia Use task See you in 10 mn! MediaEval 2016: A multimodal system for the Verifying Multimedia Use task C´edric Maigrot Vincent Claveau Ewa Kijak Ronan Sicre {firstname}.{lastname}@irisa.fr MediaEval 2016: A multimodal system for the Verifying Multimedia Use task C´edric Maigrot Vincent Claveau Ewa Kijak Ronan Sicre {firstname}.{lastname}@irisa.fr Why use a multimodal system ? Because there are several types of hoax ! » False information present in the text content » Forged image » Image reused for an other event Global Hypotheses » Prediction is first made at the image-level, then propagated to the tweets that contain the image » Translation if the detected language is different than english Text-based approach (run-T) Detect if the message is style-wise similar to known hoax § Capture similar comments between an unknown image and an image from the training set (e.g. It’s photoshopped) and similar genres of com- ments (e.g. presence of smileys) § Prediction made by a k-Nearest-Neighbor ap- proach (in this case k = 1) Source-based approach (run-S) Detect if the message is related to a trustworthy source § 2 type of sources searched: news-related organ- isms (e.g. press agencies) and explicit citations of the source of the image (e.g. the pattern pho- tographed by + Name) § Predict real if a trustworthy source is detected, fake else Example Image-based approach (run-I) Detect a known image » Compare an unknown image to an image database of 8 000 known images (7 500 fake and 500 real images) » Database images extracted from 5 specialized websites » Description of an image by a deep CNN layer output (4096-dimensional descriptor) » Predict real (resp. fake) if a real (resp. fake) similar image is found in the database, uncertain else Combination approach (run-C) Combine the three previous predictions » Late fusion: learn the best combination » Boosting algorithm (adaboost.MH, parameters of the machine learning algorithm are set by cross-validation on the training data) Results run-T run-I run-S run-C 92.23% 34.07% 94.63% 91.22% 63.98% 49.18% 90.3% 75.25% 75.57% 40.25% 92.42% 82.47% Approaches Scorein% » 2 228 messages to classify, corresponding to 130 images » 86 % to the test tweets are associated with one or more images (the rest is associated with video) Conclusion » Text-based approach: competes with the source-based approach in terms of recall but tends to classify every tweet as fake » Image-based approach: low precision compared with estimations on the training set. This may be due to: (1) small and unbalanced reference database; (2) original image and forged ones are sometimes very similar; (3) presence of stamps » Combination-based approach: does not offer any gain due to overfitting Acknowledgements This work is partly supported by the Direction G´en´erale de l’Armement, France (DGA). Maigrot, Claveau, Kijak, Sicre Linkmedia team participation: a multimodal system for the Verifying Multimedia Use taskOctober 20, 2016 3 / 3