Some days ago I was participating in a bluesky thread and a Bluesky post by Yves Venedey reminded me of a project idea I wanted to give a try: Fact Checking on bsky! You can actually extend the atproto ecosystem with a labeler, which makes these labels available to everyone who wants to see them.
Fast forward: Today I publish atproto-fact-labeler and have a live instance at @facts.kiesel.app as a BETA for you to give a try!
It's a matter of pressing "subscribe":
Similar to TopicWatchdog (which tried to extract claims from public youtube videos) I do this in my spare time and for my personal learning journey using my machine learning and federated social networks exploration.
My premise is simple: There are independent fact checkers which already publish verdicts (dpa, afp, politifact, usa today, ...) but the bsky users won't see them if they don't search on e.g. Fact Check Explorer. And sometimes they are even hidden (making people like Correctiv leave ClaimReview markup). The labeler closes this gap by making it possible to "report" (a hard word for this) any post that contains a factual claim to @facts.kiesel.app like this:
The labeler has a ML pipeline which is basically: extract, retrieve, rerank, nli and finally aggregate. And it publishes pds records for a custom lexicon to bsky, too! The schema for the lexicon is documented at kiesel-app/facts on github and will eventually be made a community lexicon once I figure how it works and if the work is appreciated by the community. After about 25 seconds it publishes the result of the process as label and quote post.
If you subscribe to the labeler it will be shown as "fact-supported" (yes the earth is round!):
For the beta to make it more visible (before somebody like the official bsky client implements it), the labeler also does a quote post like this:
It's really important to notice that the "llm" involved does not decide about true or false. It queries based on the different claim review sources available to the labeler and finds a matching one to give a verdict based on (maybe contradicting) claim review sources.
For the implementation I tried to test a lot with the earth is round and negated
versions of this claim. That was quite challenging, because the language of claims and the
fact checks themselves make specific embedding models match or not match. You can (for the atproto nerd stuff) checkout the pds app.kiesel.facts.claimVerdict at https://pdsls.dev/at://did:plc:7elfhdwxzvsqib4wmfn3zra7/app.kiesel.facts.claimVerdict/3moroxql3bq2s for the pds entry of the claim
or check the html version served by facts.kiesel.app at https://facts.kiesel.app/posts?uri=at://did:plc:7xlshkxpjrshetcx246a53yx/app.bsky.feed.post/3mois5vfgfk26:
But now it's time for a beta test and I am looking forward to your feedback either in the issues of atproto-fact-labeler concerning the implementation or for the atproto lexicon directly at https://github.com/kiesel-app/facts.
This post was written by a human. #notai
