Sistema de Ingestão de Eventos para Mapas Culturais

Problema

O Mapas Culturais de Fortaleza não possui alimentação automatizada de eventos externos. Atualmente, a cobertura de eventos é baixa, os dados são inconsistentes, há retrabalho manual na coleta e publicação, e risco de duplicidade. Eventos de plataformas de ingressos (Sympla, Bilheteria Virtual, Uhhuu, Eventim, Ticket For Fun) não são sistematicamente coletados e integrados ao Mapas.

Solução

Construir um hub de ingestão leve que:

1. Coleta automática
   Eventos de plataformas de ingressos e fontes web
2. Normalização e filtro
   Apenas eventos presenciais, em Fortaleza, nos próximos 30 dias
3. Reconciliação
   Matching de espaços com o Mapas Culturais via API
4. Curadoria
   Interface para revisão, edição e aprovação de eventos
5. Exportação
   CSV para importação no Mapas
6. Validação
   Conferência manual pós-importação

User Stories

Coleta e Ingestão

1. As a system operator, I want daily automated crawling of ticket platforms, so that events are collected without manual intervention
2. As a system operator, I want to crawl multiple platforms (Sympla, Bilheteria Virtual, Uhhuu, Eventim, Ticket For Fun), so that event coverage is comprehensive
3. As a system operator, I want to supplement platform crawling with web search for events, so that I capture events from non-platform sources
4. As a system operator, I want the system to filter only in-person events in Fortaleza within the next 30 days, so that irrelevant events are excluded
5. As a system operator, I want the system to detect and handle duplicate events from the same platform (by external_id), so that the database stays clean
6. As a system operator, I want crawling to run on a daily schedule, so that the system stays up-to-date automatically
7. As a system operator, I want to see logs of each crawling run (success/failure, event count), so that I can monitor system health
8. As a system operator, I want to receive email alerts when crawling fails, so that I can intervene quickly

Normalização e Dados

9. As a data consumer, I want event titles, descriptions, and other text fields to be cleaned (HTML removed, spaces normalized), so that data is consistent
10. As a data consumer, I want dates and times to be in a standardized format (ISO 8601, Fortaleza timezone), so that events can be compared and filtered reliably
11. As a data consumer, I want venue name and address to be extracted and stored separately, so that matching with Mapas spaces is possible
12. As a data consumer, I want to preserve the original raw payload from each platform, so that I can re-process if normalization logic changes
13. As a data consumer, I want to store event price, avatar URL, and source URL, so that the exported CSV has all necessary fields

Matching e Reconciliação

14. As a curator, I want the system to suggest Mapas spaces for each event based on venue name similarity, so that I don't have to manually search for matches
15. As a curator, I want to see the top 3 candidate spaces when the match is uncertain (60-80% similarity), so that I can make an informed decision
16. As a curator, I want to manually associate an event with a Mapas space when automatic matching fails, so that no event is left without a space
17. As a curator, I want the system to sync Mapas spaces daily, so that the candidate list is always up-to-date
18. As a curator, I want the system to mark events as "matched" (automatic), "suggested" (needs review), or "pending" (no match found), so that I can prioritize my work

Curadoria

19. As a curator, I want to see a list of all collected events with filters (platform, status, date range), so that I can find events to review efficiently
20. As a curator, I want to search events by title and description using full-text search, so that I can quickly find specific events
21. As a curator, I want to edit event fields (title, description, venue, start time, end time, price), so that I can correct errors or improve data quality
22. As a curator, I want to approve or reject events individually, so that only high-quality events are exported
23. As a curator, I want to see the source URL of each event, so that I can verify details on the original platform
24. As a curator, I want to see when an event was collected and last updated, so that I can assess data freshness
25. As a curator, I want to log in with a simple email code (magic link), so that I don't need to remember passwords

Exportação e Validação

26. As a curator, I want to generate a CSV file with all approved events that have a Mapas space assigned, so that I can import them into Mapas
27. As a curator, I want the CSV to follow the exact format required by Mapas Culturais, so that import succeeds without errors
28. As a curator, I want to download the generated CSV file, so that I can upload it to Mapas
29. As a curator, I want to see a history of past exports (date, event count, file), so that I can track what was exported
30. As a curator, I want to mark events as "imported" after successful import into Mapas, so that I can track the pipeline status
31. As a curator, I want to mark events as "error" if import failed, so that I can investigate and re-export
32. As a curator, I want to manually verify imported events via their Mapas URL, so that I can confirm they appear correctly

Dashboard e Monitoramento

33. As a system operator, I want to see dashboard stats (total events collected, pending, approved, exported, imported), so that I can understand system throughput
34. As a system operator, I want to see the status of each platform (last crawl time, success/failure), so that I can identify issues
35. As a system operator, I want to see a timeline of sync runs (Mapas sync, platform crawls), so that I can verify the system is running as expected

Administração

36. As a system operator, I want to add or deactivate platforms, so that I can control which sources are crawled
37. As a system operator, I want to manually trigger a crawl for a specific platform, so that I can test changes or recover from failures
38. As a system operator, I want to manually trigger a Mapas space sync, so that I can refresh the candidate list after changes in Mapas
39. As a system operator, I want to view and delete events, so that I can clean up test data or invalid entries

Decisões de Implementação

Stack Tecnológica

Crawling

Modelo de Dados

Matching de Espaço

Exportação CSV

Automação

Decisões de Teste

Princípios

• Testar comportamento externo, não detalhes de implementação
• Testar fluxos completos (end-to-end) para funcionalidades críticas
• Testar casos de erro e edge cases
• Usar fixtures para dados de exemplo

Módulos a Testar

1. Crawlers
   Mockar respostas HTTP, validar parsing e normalização
2. Normalizer
   Testar limpeza de texto, padronização de datas
3. Matcher
   Testar similaridade com nomes conhecidos, validar thresholds
4. CSV Exporter
   Testar formato do CSV, campos obrigatórios
5. Mapas API client
   Mockar respostas, testar retries e error handling
6. Auth
   Testar geração e validação de código magic link

Escopo

Fora do Escopo (V1)

• RabbitMQ ou outro message broker
• Meilisearch ou outro motor de busca externo
• Docker/Kubernetes
• Múltiplos usuários com permissões diferentes
• Push via API do Mapas (só CSV)
• Detecção de duplicidade entre plataformas diferentes
• Machine learning para matching
• Atualização automática de eventos já importados
• Múltiplas cidades (só Fortaleza)
• Eventos online ou híbridos (só presenciais)
• Criação automática de espaços no Mapas
• Webhooks ou notificações em tempo real
• Dashboard com gráficos (só números)

Pode Entrar se Sobrar Tempo

• Busca web complementar (eventos fora de plataformas conhecidas)
• Dashboard com gráficos básicos
• Exportação em outros formatos (JSON, XML)
• Webhook para notificar curadores de novos eventos
• Logs mais detalhados (audit trail de edições)

Notas Adicionais

Princípios de Design

• Código mínimo, valor máximo: cada linha de código deve justificar sua existência
• SQLite até provar que não aguenta: não adicionar complexidade de banco desnecessária
• Automatizar só depois de manual funcionando: validar fluxo manualmente antes de automatizar
• Curadoria humana é essencial: não tentar automatizar tudo, deixar decisões críticas para humanos

Riscos

Métricas de Sucesso

• Cobertura: >80% dos eventos das plataformas principais coletados
• Qualidade: >90% dos eventos exportados importados com sucesso no Mapas
• Velocidade: crawling completo em <30 minutos
• Uso: curador consegue revisar 50 eventos/hora

Próximos Passos