Public documentation
Methodology & Limitations
This page documents exactly how a record gets from a published news story to a marker on the map, what the confidence score means, and what conclusions this dataset can and cannot support. It is written to be checkable: every claim below corresponds to code in the public repository.
Read this first
- Every record describes an allegation as reported. Nothing here is a finding of guilt, and an entry must not be treated as evidence that any person committed an offence.
- Records are de-identified in two passes. The extraction prompt instructs the model to omit victim and accused identities, minors and exact addresses; a separate deterministic pass then strips phone numbers, email addresses and ID numbers from every record regardless of how it was produced. Neither pass can guarantee removal of personal names — the first is an instruction to a model, and no pattern match reliably identifies names in Bengali and English prose. If you find identifying information in a record, report it and it will be removed.
- This site is not a reporting channel and is not monitored. In an emergency call 999.
The ingestion pipeline
1 · Collection
A scheduled job runs every six hours and reads the public RSS feeds of major Bangladeshi dailies in English and Bengali, plus the Criminal Investigation Department's official press-release feed. A keyword gate discards anything with no crime signal before it costs a model call. Official sources publish far less often than newsrooms, so they are given reserved slots — otherwise every police release would be pushed below the per-run cap and the archive would be entirely newswire.
2 · Extraction
Surviving text goes to Gemini Flash under a fixed JSON schema. The model reads English, Bengali and Banglish, and returns a neutral English title and summary, a category, a date, a thana, and any statute the source actually cites.
3 · Normalisation
The model is never trusted for geography. It names a place; a gazetteer resolves that name — through aliases, Bengali spellings and fuzzy matching — to one of the 50 Dhaka Metropolitan thanas, or failing that to one of the country's 64 districts. A place that resolves to neither is discarded, because a guessed location is worse than a missing record.
4 · Deduplication
Each record is keyed by SHA-256 over (incident date, thana, title). Re-running the same window inserts nothing new, and outlets whose headlines normalise alike collapse to one row. Headlines that differ — a Bengali one and an English one, say — do not collapse, so a heavily covered incident can appear more than once.
5 · Publication
Records land in PostgreSQL and are served to this site through a session-authenticated API. Nothing is hand-edited between extraction and publication; corrections are made by changing the pipeline or removing the record.
Confidence scoring
Confidence is a property of the source, not of the model's certainty. It is fixed by policy at ingestion time and is never adjusted per record — so a score tells you where a record came from, and nothing more.
| Tier | Score | Source class | What it means |
|---|---|---|---|
| Official | 95 | Public posts from police or government channels | The originating authority is speaking directly. Still a claim about an allegation, not a court finding. |
| Verified News | 80 | Established Bangladeshi news outlets with a named masthead | Reported by journalists working to an editorial standard, usually citing police sources. |
| Public Social | 55 | Publicly visible social-media posts | Fastest signal, weakest verification — treat as a lead to check, not an established fact. Currently unpopulated: the collector needs a Meta Page access token, and the public RSS mirrors it falls back to now refuse automated requests. |
The “verified source ratio” on the dashboard is the share of records scoring 80 or above — that is, everything except public social media.
What this dataset cannot tell you
- This is a record of what was reported, not of what happened. Offences that never reach a newsroom or an official feed do not appear here, and under-reporting is not evenly distributed across offence types or neighbourhoods.
- Coverage intensity is not crime intensity. A thana that receives more press attention will show more incidents than a comparable thana that receives less, independent of actual crime rates.
- Locations are centroids, not scenes. Inside the Dhaka Metropolitan Police area a marker sits at the centre of the responsible thana; elsewhere in the country only district-level precision is available, so the marker sits at the district centre and may be many kilometres from the actual location. Never read a pin as a street address.
- An incident whose location cannot be resolved to a known thana or district is discarded rather than placed. Records are therefore missing, not misplaced — but reporting that names no recoverable location will not appear here at all.
- Categories are assigned by a language model. It is accurate on clear-cut reporting and less so on ambiguous or partial reports; a small share of records will be miscategorised.
- Dates default to the publication date when the source does not state when the incident occurred, and to midnight BST when it gives a date but no time.
- Reports about old cases are dated to the announcement, not the offence. A police release about arrests in an eight-year-old murder enters the archive on the date of the release, because the pipeline cannot reliably tell a genuine historical date from a year the model invented. Read such records as news of an arrest, not of a fresh incident.
- The archive begins when this system started collecting. It is not a historical series and cannot support year-over-year comparison across that boundary.
- Deduplication is keyed on the headline, so a widely covered incident reported under different headlines — particularly across Bengali and English outlets — can appear as more than one record and inflate counts for that incident.
Legal and ethical position
Sources. Only material already published for the general public is collected: RSS feeds offered by outlets for syndication, and posts published openly by official accounts. No private group, closed account, authenticated surface, or leaked material is read, and no login or cookie is replayed to obtain content.
Attribution. Every record links to the original report. Summaries are short, factual restatements written to be checked against the source, not substitutes for reading it. Full article text is not republished.
Privacy. Contact details and identifiers — phone numbers, email addresses, national ID and passport numbers — are stripped from every record by a deterministic pass that does not depend on the model. Names, addresses and details about minors are excluded by instruction to the extraction model, which is weaker: an instruction can be missed. Where the model is unavailable and the pipeline falls back to keyword classification, no article-derived summary is published at all, precisely because there is then nothing enforcing redaction.
Presumption of innocence. Bangladeshi law presumes innocence until conviction. Records here report accusations at a moment in time; charges are frequently amended, withdrawn, or dismissed, and this archive does not track those outcomes.
Corrections and removal. To report an inaccurate record, or to request removal of one that names or identifies a person, open an issue on the project repository with the record ID shown in the archive. Records that identify an individual are removed on request without argument.
Not professional advice. This portal is a research and transparency tool. It is not a security assessment, and it must not be used to make decisions about an individual — in hiring, tenancy, lending, or anything else.
Access to the data
The compiled dataset is not published as an open API or a bulk download. Reading it through this site requires a session, and the interactive API documentation is not served.
That is a deliberate limit on redistribution of the compiled product, not an attempt to obscure the underlying facts. Every record carries a link to the original report, which is public and freely readable, so any individual claim on this site can still be checked against its source.
It is also worth being plain about the limit of the measure: anything a web page renders can be retrieved by someone determined enough. This raises the cost of casual scraping; it is not a guarantee of exclusivity.