How do you handle document chunking and metadata filtering when dealing with complex high-value documents that contain mixed tables and text?

For complex documents with tables and narrative text, I use a multi‑step chunking pipeline. First, I split the PDF into logical sections using a layout parser (e.g., pdfplumber or Camelot) to isolate tables, headings, and paragraphs. Each section is then tokenized into chunks of 500–800 tokens with a 50‑token overlap to preserve context. I generate embeddings for each chunk with a domain‑tuned model (e.g., Sentence‑BERT or a custom fine‑tuned transformer). Metadata is extracted during parsing: document ID, section title, table ID, and any key fields (e.g., dates, amounts). I store chunks and metadata in a vector store (Pinecone, Weaviate) and index the metadata as searchable attributes. During retrieval, I first filter by metadata (e.g., only tables from a specific report) to reduce the candidate set, then perform vector similarity search on the filtered set. This two‑stage approach keeps the search efficient while ensuring that only relevant, high‑value chunks are considered.