Module 2

Vendor Evaluation Framework

How to systematically evaluate payment providers using structured criteria and run proof of concepts that actually matter

---

What Vendor Evaluation Is Really About

Vendor evaluation for payments is rarely objective. Most evaluations are driven by sales conversations, RFP responses that promise everything, and demonstrations that show the vendor's product in the best possible light. You end up choosing a vendor based on who has the best sales team, not who best solves your problem.

The framework in this module fixes that. It forces you to distinguish between what vendors claim matters and what actually matters for your business. It gives you a structured way to compare vendors on apples-to-apples criteria. It tells you exactly what to test in a proof of concept and what success actually looks like. Most importantly, it forces you to quantify the trade-offs. No vendor is perfect. Every vendor makes trade-offs between cost, features, reliability, and support. Your job is to understand the trade-offs and choose the one that aligns with your priorities.

Vendor evaluation fails when organisations optimise for the wrong criteria. Sales teams optimise for procurement criteria (SLAs, compliance certifications, references). Operations teams optimise for reliability and support. Product teams optimise for feature coverage. Finance teams optimise for cost. You need a framework that lets each stakeholder weight their priorities, then choose the vendor that best optimises the total.

The Vendors You Are Evaluating

Payment vendor categories have specialised. You are probably evaluating multiple vendors in different categories. A single "payment processor" no longer exists. Instead, you evaluate gateways (Stripe, Adyen), fraud and risk providers (Sift, Kount), identity verification providers (Socure, IDology), acquiring partners (Worldpay, Global Payments), and orchestration layers. This module applies to all of them, but with different criteria.

For a gateway vendor, you weight technical capability heavily. For a fraud vendor, you weight model accuracy and coverage heavily. For an acquirer, you weight cost and geographic coverage heavily. The framework is the same, but the specific criteria and weights change.

---

The Evaluation Criteria That Actually Matter

Technical Capability

Technical capability means: can this vendor handle your transaction types, payment methods, and geographic requirements? Can they integrate cleanly into your architecture? How much engineering effort is required to integrate? How much operational burden do they create?

Evaluate on these sub-criteria:

• API design quality: Is the API well-documented? Can you understand the request and response format by reading the docs, or do you need to contact support? Are there SDKs in your language stack? Are the SDKs well-maintained? Does the API use modern patterns (REST, JSON, webhooks) or legacy approaches (XML, SOAP)? Test this by integrating a simple charge endpoint. Time how long it takes to get a working integration running. Good API design gets working integrations in under 4 hours. Poor API design takes 2 weeks.
• Payment method coverage: Does the vendor support all payment methods you need? If you support 12 payment methods but the vendor only supports 8, you need a secondary vendor to fill the gap. Coverage should match your transaction mix.
• Geographic coverage: Does the vendor support acquiring in all geographies you need? Do they support local payment methods? Do they support local currency settlement? A vendor with strong North American presence but weak Asia-Pacific presence is not suitable if your customer base is global.
• Latency: How long does authorisation take? Median latency should be under 500ms. P95 latency should be under 1 second. Higher latency degrades user experience.
• Uptime SLAs: What uptime does the vendor guarantee? 99.5 percent is baseline. 99.9 percent is good. 99.95 percent is excellent. But SLAs are only meaningful if backed by real-world performance monitoring and if they include service credits for violations.

Commercial Terms

Commercial terms determine your cost structure and flexibility. Evaluate on these sub-criteria:

• Pricing model: Interchange-plus (IC+) is the most transparent model: you pay actual interchange plus a fixed markup. Blended or tiered pricing obscures the true cost. Understand which model the vendor uses and why it matters for your volume and transaction mix.
• Volume commitments: Does the vendor require volume commitments? If yes, what happens if you fall short? What discounts do you get for hitting commitments? Are commitments on transaction count, transaction value, or both?
• Contract flexibility: How long is the contract? 12 months? 36 months? Can you terminate early if the vendor fails to meet SLAs? What are the termination fees? A 36-month contract with no early exit is a red flag.
• Fee transparency: Does the vendor clearly disclose all fees? Hidden fees (monthly minimums, batch fees, exception handling fees, PCI compliance fees) add up quickly. Ask for a sample invoice and an itemised cost breakdown.
• Negotiation leverage: Is the vendor's pricing standardised or negotiable? If you are a large merchant (over $10 million annual volume), you should be able to negotiate lower rates. If the vendor refuses to negotiate, it signals they view you as commoditised.

Operational Support

Operational support determines how quickly you can resolve problems and how much engineering time you spend firefighting. Evaluate on these sub-criteria:

• Integration support: Does the vendor provide hands-on integration support, or is support purely self-serve documentation? Good vendors have technical integration specialists who help you get live faster. Poor vendors leave you to figure it out from docs.
• Incident response: When something goes wrong (processor outage, API bug, unusual activity), how quickly does the vendor respond? Do they have on-call support? What are response time SLAs? For critical issues, response within 1 hour is baseline. 15 minutes is good.
• Account management: Do you get a dedicated account manager? Are they proactive about notifying you of changes, deprecated features, or performance issues? Or are account relationships purely transactional?
• Communication quality: When you email support, how long until you get a response? Is the response helpful or does it require multiple back-and-forths? Support quality varies wildly between vendors.
• Documentation: Is the documentation comprehensive and searchable? Are there code examples? Are there troubleshooting guides? Poor documentation creates support overhead.

Compliance and Security

Compliance and security determine your risk exposure and operational burden. Evaluate on these sub-criteria:

• PCI compliance scope reduction: How much of PCI compliance does the vendor handle? Can you use their tokenisation to reduce your PCI scope to SAQ-A? Or does using their service expand your PCI scope? This matters for audit cost and operational burden.
• Data residency: Where is your transaction data stored? Can you specify data residency requirements (GDPR, CCPA, data localisation laws)? Can the vendor guarantee that payment data never leaves specified regions?
• Certifications: Does the vendor have relevant certifications? PCI-DSS Level 1, SOC 2 Type II, ISO 27001. These are baseline. Check when they were last audited.
• Regulatory licences: Does the vendor have necessary regulatory licences? If they operate as a payment processor, they need Money Transmitter licences. If they operate as an acquirer, they need acquiring licences. Unlicensed vendors create regulatory risk for you.
• Audit availability: Will the vendor submit to third-party audits of their systems? If you are a large enterprise customer, you may need to audit the vendor's infrastructure. Vendors that refuse audits are problematic.

Reliability and Scalability

Reliability and scalability determine whether the vendor can handle your growth and whether your business survives processor outages. Evaluate on these sub-criteria:

• Disaster recovery: Does the vendor have a disaster recovery plan? Where are backups stored? How long to restore? Can they operate from a secondary data centre if the primary goes down?
• Redundancy: Does the vendor route your traffic through multiple data centres? Or do they route through a single point of failure?
• Scaling headroom: As your transaction volume grows, does the vendor's infrastructure scale transparently? Have they ever turned away merchants due to capacity constraints?
• Performance monitoring: Does the vendor expose performance metrics (latency percentiles, success rates, error rates)? Can you see real-time dashboards or does the vendor only report monthly summaries?

---

Running a Proof of Concept That Matters

A proof of concept is the opportunity to test vendor claims against your real environment. Many POCs fail because they test the wrong things or run for too long without clear success criteria. A good POC is time-bounded (2 to 4 weeks maximum), focused on critical decision points, and has defined success criteria before you start.

What to Test

Your POC should test the criteria that differentiate vendors. If all vendors have similar API quality and coverage, do not spend 2 weeks testing API quality. Test what matters. For a multi-currency payment processor, test: local currency support in each geography, settlement accuracy, reconciliation automation. For a fraud vendor, test: fraud catch rate on your transaction mix, false decline rate, and model accuracy on your specific verticals.

POC Timeline and Scope

A tight POC follows this structure: Week 1 is integration and basic transaction testing. Week 2 is edge case testing (failures, exceptions, unusual transactions). Week 3 is performance testing and volume scaling. Week 4 is refining based on learnings and making the decision. If a vendor cannot meet the basic success criteria by end of week 1, stop the POC. Do not throw good time after bad.

Success Criteria

Define success criteria in writing before the POC starts. Example success criteria for a payment gateway POC might be:

• Complete API integration in under 20 hours of engineering time.
• Process 1,000 test transactions with 100 percent success rate.
• Latency under 500ms for 99 percent of transactions.
• Webhook delivery with less than 1 second latency.
• Reconciliation matches between your system and vendor system with zero discrepancies after 3 days of transactions.
• Support response time under 4 hours for integration questions.

These are specific and measurable. If the vendor does not meet all criteria, you know to keep evaluating.

Total Cost of Ownership Modelling

Per-transaction fees are visible, but TCO is what matters. TCO includes visible costs (processing fees) plus hidden costs (integration maintenance, reconciliation overhead, compliance overhead, support time). A cheap vendor with poor API design might cost more in total engineering time than an expensive vendor with excellent API design.

Model TCO like this. Start with processing fees: your annual transaction volume times your effective rate. Add integration costs: the engineering hours to integrate the vendor times your loaded hourly cost. Add operational overhead: the monthly engineering hours spent on reconciliation, monitoring, and support. Add compliance costs: annual audit costs, remediation labour. Add vendor switching costs if applicable: the cost to migrate if this vendor does not work out.

A TCO model might look like: Vendor A costs $250,000 in annual processing fees, $40,000 in integration, $15,000 in operational overhead = $305,000 total. Vendor B costs $210,000 in processing fees, $80,000 in integration, $8,000 in overhead = $298,000 total. On raw fees, Vendor A looks better. On TCO, Vendor B is slightly cheaper.

---

Red Flags in Vendor Evaluations

Certain vendor behaviours signal deeper problems. Watch for these red flags:

• Vague pricing: If the vendor cannot clearly explain their pricing model or says "we will negotiate after you sign," that is a red flag. Pricing should be transparent upfront.
• No SLA or SLA with no service credits: An SLA without service credits is theatre. It means nothing if the vendor violates it and faces no penalty. Insist on service credits if the vendor misses SLAs.
• Slow integration support: If the vendor cannot get you moving in integration within the first week, they do not have adequate support. Their integration team is undersized or not incentivised to help.
• Unwillingness to run a POC: If the vendor refuses to let you run a POC or insists on a long, expensive POC, they are hiding something. Good vendors are confident enough to let you test.
• Limited API documentation: Poor documentation is a proxy for poor engineering. If the docs are incomplete, the API is probably poorly designed.
• No real-time monitoring: If you cannot see real-time transaction status or performance metrics, the vendor is not transparent about their operations.
• Lock-in contracts: 36-month contracts with no early exit clause are red flags. You should be able to leave if the vendor fails to meet commitments.
• No roadmap or long-term vision: If you ask about the vendor's roadmap and they cannot articulate a clear vision, they may not be stable long-term.

---

Connecting Vendor Evaluation to the Major Matters AI Tools Directory

The Major Matters AI Tools Directory reviews fraud and risk vendors using similar criteria to this framework. When you evaluate a fraud vendor for your own business, read the Directory review. See what criteria the Directory weighted. See what issues other implementers ran into. This cross-reference helps you avoid surprises.

Vendor evaluation is never finished. The market changes. Vendors ship new features or degrade existing ones. Revisit your vendor evaluation annually. Update your scorecard. Stay honest about whether your choice still makes sense or whether you should start evaluating alternatives.