RAG Done Right: A Practical Architecture for Australian Startups

Retrieval-Augmented Generation (RAG) is everywhere in startup pitch decks right now. The promise is simple: connect an LLM to your data, and suddenly you’ve got an AI product that knows your business. The reality is messier. Most RAG implementations fail quietly-slow responses, hallucinated answers, or costs that spiral. We’ve rebuilt enough broken RAG systems to know where founders go wrong, and we’ve learned what actually works for Australian teams shipping on real timelines.

This is not a tutorial on how RAG works in theory. This is what we’ve learned building it in production, and the architecture decisions that separate a prototype from something you’d trust with customer data.

Why Most RAG Projects Fail (And It’s Usually Not the LLM)

The failure pattern is consistent: a founder builds a RAG pipeline, tests it on a small dataset, and it works great. They add more documents. Performance tanks. Costs blow out. The LLM starts making up answers that sound plausible but are completely wrong. They blame the model. The problem was usually never the model.

RAG failure almost always comes down to retrieval quality. If you’re pulling the wrong documents from your vector database, no LLM-no matter how smart-can fix that. The garbage is already in the prompt. You’re asking the model to hallucinate because the facts aren’t there.

The second failure point is operational: the system works fine with 100 documents and breaks with 10,000. Vector database costs grow linearly with data volume. Retrieval latency increases. Your prompt becomes too long and tokens start getting cut off. You never tested it at scale, so production becomes your QA environment.

Third: people underestimate the complexity of indexing. Your data isn’t clean. PDFs have weird formatting. Tables get mangled. Text is duplicated. You need to build-or buy-a real data pipeline. A simple script that dumps text into a vector database will fail you.

The Architecture That Actually Works

Here’s the stack we use for Australian startups, and why each component matters:

1. Data ingestion layer (not just a loop): Use something like Unstructured.io or LlamaIndex to properly parse your source material. Different document types need different handling. A CSV is not a PDF. PDFs with tables need special care. Budget 3-4 weeks just for this if you’re starting with legacy documents.
2. Chunking strategy (not random splits): Don’t chunk by token count alone. Chunk by semantic meaning. A paragraph about pricing should stay together even if it’s 500 tokens. A 2000-token legal section should be split. We typically use 400-600 token chunks with 100-token overlap. Test this number against your actual queries.
3. Vector database (managed, not self-hosted): Pinecone or Weaviate managed, not trying to run PostgreSQL + pgvector on your own infrastructure. The operational overhead kills most teams. You’ll pay roughly 200-500 AUD per month for a production vector DB. That’s not an expense-that’s clarity. Self-hosting buys you a DevOps problem.
4. Hybrid search (vector + keyword): Vector search is great for semantic relevance. Keyword search is great for exact matches. Use both. A query for “APRA rules 2024” needs keyword matching; a query about sentiment in earnings calls needs vectors. Most production systems use a weighted combination (we typically weight vectors at 0.7, keyword at 0.3, then adjust).
5. Re-ranking layer (this is the secret): After you retrieve your top 10 documents, re-rank them with a smaller, faster model (like Cohere Rerank 3 or a cross-encoder). This costs a fraction of the LLM call but dramatically improves quality. Don’t skip this. It’s the difference between usable and frustrating.
6. LLM orchestration (structured outputs): Use Claude or GPT-4 with structured output modes. Force the model to return JSON with confidence scores and source citations. This isn’t pretty-it’s essential. You need to know which documents the answer came from, and you need to programmatically know whether the model is confident.

Real Numbers: Cost and Latency

This is where most founders get surprised. Let’s say you’re building a customer support AI for a mid-market SaaS with about 500 documents (10,000 pages of documentation, help articles, and FAQs). Realistic monthly costs:

• Vector database: 300 AUD
• LLM calls (assuming 500 queries per day, average 2000 tokens output): 400-600 AUD
• Re-ranking and embeddings: 150 AUD
• Infrastructure (API layer, monitoring): 200 AUD
• Total: roughly 1050-1350 AUD per month at 500 daily queries

At 5000 daily queries, you’re looking at 8000-12,000 AUD per month. Most founders don’t price this correctly and end up shocked when the invoice lands.

Latency: a well-built RAG system should respond in 2-4 seconds end-to-end (retrieval + re-ranking + LLM generation). If you’re hitting 8+ seconds, your bottleneck is usually the LLM generation time, which means you need shorter context windows. This is where prompt engineering matters-every token costs money and time.

The Implementation Reality: 28-Day MVP vs. Production

We ship RAG MVPs in 4 weeks. Here’s what that looks like:

Week 1: Data ingestion and chunking. You’ll spend time understanding your source data. It’s always worse than you think. Corrupted files, mixed encodings, tables that don’t parse.

Week 2: Vector database setup and initial indexing. Get documents in. Run basic retrieval tests against your queries. Measure retrieval quality (precision and recall matter more than you’d think).

Week 3: LLM integration, prompt tuning, and re-ranking. This is where the actual product comes together. Test with real users or at least stakeholders. Most prompts need 15-20 iterations.

Week 4: Polish, monitoring, cost optimisation, and deployment. Add logging so you can see which queries are failing. Set up alerts. Document everything.

What you won’t do in 4 weeks: build a perfect data pipeline, handle every edge case, or optimise for thousands of concurrent users. You’ll have a working system that solves the core problem. That’s the goal.

If you’re serious about shipping a real product and not just a prototype, talk to Amora about your build. We’ve done enough of these to know the shortcuts and the traps.

Common Mistakes to Avoid

Mistake 1: Ignoring data quality. You can’t RAG your way out of bad source documents. Spend time cleaning. It’s boring. It’s necessary.

Mistake 2: Testing on toy data. Your demo dataset of 50 documents will work beautifully. Your production dataset of 50,000 won’t. Always test at scale before launch.

Mistake 3: Not monitoring retrieval. Add logging to see what documents are actually being retrieved for each query. Most teams never look at this and don’t realise their retrieval is terrible.

Mistake 4: Chunking too small. 128-token chunks sound efficient until you realise you’ve lost all context. You end up retrieving 20 documents to answer one question. Start with 400-600 and adjust.

Mistake 5: Forgetting about versioning. Your documents will change. Your chunks will change. Your embeddings will change. You need to version your index and be able to rollback. Version control your data like you version your code.

When RAG Is Actually the Right Answer

RAG works brilliantly for specific problems:

• Customer support (answering questions about your product or docs)
• Internal knowledge bases (helping staff find policies, procedures, past decisions)
• Document analysis (summarising, extracting information, finding patterns across many files)
• Fact-grounded chat (anything where hallucination is unacceptable)

It doesn’t work well for:

• Reasoning problems that require calculation or logic
• Multi-step workflows that need to act on external systems
• Content generation where creativity matters more than accuracy

Know which bucket your problem falls into before you start building.

The Next Step

If you’re a founder evaluating whether to build RAG into your product, start with a tight scope. Pick one specific use case. Build the infrastructure right the first time-that means proper chunking, hybrid search, re-ranking, and monitoring. Get it in front of users in 4 weeks. Measure whether it actually solves a problem better than the alternative (which is often just a search box). Then decide whether to expand.

RAG is powerful. It’s just not magic. The teams that win with it are the ones who understand the architecture, respect the costs, and test relentlessly before shipping.