liqiang/ai-agent-book
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

ai-agent-book 精选快照（<2MB 代码与文档，来自 github.com/bojieli/ai-agent-book）
Build latest book artifacts / build (push) Canceled after 0s
Details
dependency resolution / resolve (3.11) (push) Canceled after 0s
Details
dependency resolution / resolve (3.13) (push) Canceled after 0s
Details
deploy-pages / build (push) Canceled after 0s
Details
deploy-pages / deploy (push) Canceled after 0s
Details
i18n consistency check / check (push) Canceled after 0s
Details
provider adoption tests / test (chapter2/context-compression) (push) Canceled after 0s
Details
provider adoption tests / test (chapter2/prompt-injection) (push) Canceled after 0s
Details
provider adoption tests / test (chapter2/system-hint) (push) Canceled after 0s
Details
provider adoption tests / test (chapter3/log-sanitization) (push) Canceled after 0s
Details
web-search-agent tests / test (push) Canceled after 0s
Details
web-search-agent tests / agentbook (push) Canceled after 0s
Details

Browse Source
This commit is contained in:
liqiang
2026-08-20 13:12:50 +00:00
commit b119135836
10275 changed files with 3284984 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
chapter3/user-memory-evaluation/test_cases/layer1/01_bank_account_setup.yaml
+217
View File
@@ -0,0 +1,217 @@
test_id: layer1_01_bank_account
category: layer1
title: Bank Account Setup - Personal Details Retrieval
description: Test agent's ability to recall customer's personal and financial information provided during a lengthy bank account
setup call
conversation_histories:
- conversation_id: bank_setup_001
timestamp: '2024-11-15 10:30:00'
metadata:
business: First National Bank
department: New Accounts
call_duration: 47 minutes
messages:
- role: user
content: Hi, I'd like to open a new checking account please.
- role: assistant
content: Good morning! I'd be happy to help you open a new checking account with First National Bank. My name is Sarah.
Before we begin, may I have your full legal name as it appears on your driver's license?
- role: user
content: Sure, it's Michael James Robertson.
- role: assistant
content: Thank you, Mr. Robertson. And could you please provide your date of birth?
- role: user
content: March 15th, 1985.
- role: assistant
content: Perfect. Now I'll need your current residential address.
- role: user
content: I live at 1847 Maple Street, Apartment 3B, Portland, Oregon, 97205.
- role: assistant
content: Thank you. How long have you been at this address?
- role: user
content: About two and a half years now.
- role: assistant
content: Great. And what's the best phone number to reach you?
- role: user
content: My cell is 503-555-8924.
- role: assistant
content: Excellent. Do you have an email address for account notifications?
- role: user
content: Yes, it's mrobertson85@email.com
- role: assistant
content: Thank you. Now, for our records, what's your current employment status?
- role: user
content: I'm employed full-time as a software engineer at TechCorp Industries.
- role: assistant
content: How long have you been with TechCorp Industries?
- role: user
content: Coming up on four years this December.
- role: assistant
content: That's wonderful. Could you provide your annual income range for our application?
- role: user
content: I make about $125,000 a year, before taxes.
- role: assistant
content: Thank you for that information. Now, I need to verify your identity. Could you provide your Social Security number?
- role: user
content: Sure, it's 547-82-9163.
- role: assistant
content: Thank you. And your driver's license number?
- role: user
content: It's Oregon license, number D758392.
- role: assistant
content: Perfect. Now, which type of checking account were you interested in? We have our Basic Checking with no minimum
balance, our Premium Checking with a $2,500 minimum, or our Elite Checking with a $10,000 minimum.
- role: user
content: What are the benefits of the Premium one?
- role: assistant
content: The Premium Checking includes no monthly fees if you maintain the minimum balance, free checks, free wire transfers
domestically, and you earn 0.5% APY on your balance.
- role: user
content: Hmm, actually let me think about it. What about the Elite?
- role: assistant
content: The Elite Checking offers everything in Premium plus 1.2% APY, free international wire transfers, a complimentary
safe deposit box, and access to our financial advisory services.
- role: user
content: You know what, let's go with the Premium. I can maintain that $2,500 minimum.
- role: assistant
content: Excellent choice. Now, would you like to order checks with your account?
- role: user
content: Yes, I'll need checks.
- role: assistant
content: Would you like the standard design or would you like to choose from our custom designs? Custom designs have a
$15 fee.
- role: user
content: Just the standard is fine.
- role: assistant
content: Perfect. We'll send those to your Maple Street address. Now, would you like to set up online banking?
- role: user
content: Absolutely, yes.
- role: assistant
content: Great. You'll need to create a username. What would you like to use?
- role: user
content: How about MRobertson503?
- role: assistant
content: Let me check... Yes, that username is available. You'll create your password when you first log in. Would you
also like to enroll in mobile banking?
- role: user
content: Yes, definitely.
- role: assistant
content: Excellent. Now, would you like to add a debit card to this account?
- role: user
content: Yes, I'll need a debit card.
- role: assistant
content: Would you like to set up a custom PIN now or when you receive the card?
- role: user
content: I'll set it up now. Can I use 4827?
- role: assistant
content: Yes, 4827 is set as your PIN. Please remember this number. Now, for overdraft protection, would you like to enroll?
- role: user
content: What are my options?
- role: assistant
content: We can link it to a savings account, a credit card, or you can opt for our standard overdraft coverage which
allows transactions to go through for a $35 fee.
- role: user
content: Actually, let me also open a savings account and link them.
- role: assistant
content: Perfect! Our basic savings requires a $100 minimum balance. Is that acceptable?
- role: user
content: Yes, that works.
- role: assistant
content: Great. I'll set that up for you. Now, for your initial deposit to open both accounts, what amount would you like
to deposit?
- role: user
content: I'll start with $5,000 in checking and $500 in savings.
- role: assistant
content: Excellent. How would you like to make this initial deposit?
- role: user
content: I can do an electronic transfer from my current bank, Wells Fargo.
- role: assistant
content: Perfect. I'll need your Wells Fargo account number and routing number.
- role: user
content: The account number is 8847293001 and the routing number is... let me check... 121000248.
- role: assistant
content: Thank you. I've initiated that transfer. It should complete within 2-3 business days. Now, would you like to
set up any automatic transfers between your checking and savings?
- role: user
content: Yes, can we do $200 monthly from checking to savings?
- role: assistant
content: Certainly. Which day of the month would you prefer?
- role: user
content: The 15th of each month would be good.
- role: assistant
content: Perfect, that's set up. Now, do you have any direct deposits you'd like to set up?
- role: user
content: Yes, my paycheck from TechCorp.
- role: assistant
content: I can provide you with our routing number and your new account number for that. Our routing number is 123006800,
and your new checking account number is 4429853327.
- role: user
content: Let me write that down... 4429853327, right?
- role: assistant
content: Correct. And your savings account number is 4429853328.
- role: user
content: Got it. Oh, I should mention, I travel internationally for work sometimes. Will my debit card work overseas?
- role: assistant
content: Yes, your debit card will work internationally. We charge a 3% foreign transaction fee. Would you like to add
travel notifications to your profile?
- role: user
content: Not right now, but good to know.
- role: assistant
content: Of course. You can always add them through online banking or by calling us. Now, for security questions for your
account, I need to set up three. First, what was the name of your first pet?
- role: user
content: Buddy. He was a golden retriever.
- role: assistant
content: 'Thank you. Second question: What city were you born in?'
- role: user
content: Denver, Colorado.
- role: assistant
content: 'And third: What was your mother''s maiden name?'
- role: user
content: Harrison.
- role: assistant
content: Perfect. Now, would you like to receive paper statements or electronic statements?
- role: user
content: Electronic is fine. Save some trees.
- role: assistant
content: Great choice! They'll be sent to mrobertson85@email.com. Is there anything else you'd like to add to your account?
- role: user
content: Actually, what about a credit card?
- role: assistant
content: We have several options. Our CashBack card gives 2% on all purchases, our Travel Rewards card gives 3x points
on travel and dining, and our Basic card has no annual fee.
- role: user
content: What's the annual fee on the CashBack card?
- role: assistant
content: The CashBack card has a $95 annual fee, but it's waived for the first year.
- role: user
content: You know what, I'll think about the credit card and maybe apply later.
- role: assistant
content: No problem at all. You can apply anytime through online banking or by calling us. Your debit card will arrive
in 7-10 business days at your Maple Street address.
- role: user
content: Perfect. Oh wait, I forgot to ask - are there any ATM fees?
- role: assistant
content: With your Premium Checking, you get unlimited free withdrawals at First National Bank ATMs, and we reimburse
up to $15 per month in other bank ATM fees.
- role: user
content: That's great. And how many First National ATMs are there in Portland?
- role: assistant
content: We have 47 ATMs throughout the Portland metro area, plus you can use any AllPoint ATM nationwide for free - that's
over 55,000 ATMs.
- role: user
content: Excellent. I think that covers everything I needed.
- role: assistant
content: 'Wonderful! Just to confirm everything: You''ve opened a Premium Checking with account number 4429853327 and
a Basic Savings with account number 4429853328. Your debit card with PIN 4827 will arrive in 7-10 business days. You''ve
set up a $200 monthly transfer on the 15th, and your online banking username is MRobertson503. Is everything correct?'
- role: user
content: Yes, that all sounds right. Thank you so much for your help, Sarah.
- role: assistant
content: You're very welcome, Mr. Robertson! If you have any questions, don't hesitate to call us at 1-800-555-BANK. Welcome
to First National Bank!
user_question: What was my checking account number again? I need it to set up my direct deposit at work.
evaluation_criteria: |
The agent should accurately retrieve and provide the specific checking account number 4429853327 that was mentioned during the account setup conversation. Since the user is asking about setting up direct deposit at work, the agent should ideally also mention the routing number 123006800 which was discussed in that context. The agent must correctly distinguish between the checking account (4429853327) and the savings account (4429853328) numbers, providing only the checking account number as requested. The response demonstrates proper memory recall if it provides the exact checking account number without confusing it with other numbers mentioned in the conversation such as the SSN, phone number, or savings account number.
expected_behavior: The agent should directly retrieve and provide the checking account number 4429853327, possibly also mentioning
the routing number 123006800 since the user is asking for direct deposit setup purposes
chapter3/user-memory-evaluation/test_cases/layer1/02_insurance_claim.yaml
+242
View File
@@ -0,0 +1,242 @@
test_id: layer1_02_insurance_claim
category: layer1
title: Auto Insurance Claim - Policy and Incident Details
description: Test agent's ability to recall specific claim details and policy information from an extensive insurance claim
call
conversation_histories:
- conversation_id: insurance_claim_001
timestamp: '2024-11-18 14:15:00'
metadata:
business: SafeGuard Auto Insurance
department: Claims Department
call_duration: 52 minutes
messages:
- role: user
content: Hi, I need to file a claim for an accident I was just in.
- role: assistant
content: I'm sorry to hear about your accident. I'm James from SafeGuard's claims department. First, are you safe and
have you received any necessary medical attention?
- role: user
content: Yes, I'm okay. Just shaken up. The paramedics checked me out at the scene and said I was fine.
- role: assistant
content: I'm glad you're alright. Let me pull up your account. Can you provide your policy number?
- role: user
content: It's SGA-8847291-04.
- role: assistant
content: Thank you. I see you're Jennifer Martinez with a 2019 Honda Accord, is that correct?
- role: user
content: Yes, that's right.
- role: assistant
content: And can you verify your date of birth for security?
- role: user
content: July 22nd, 1988.
- role: assistant
content: Perfect, thank you. Now, can you tell me when and where this accident occurred?
- role: user
content: It happened about two hours ago, around 12:15 PM, at the intersection of Broadway and 45th Street in Sacramento.
- role: assistant
content: Were the police called to the scene?
- role: user
content: Yes, Officer Bradley took the report. The report number is SAC-2024-78432.
- role: assistant
content: Excellent, having that police report will help. Can you describe what happened?
- role: user
content: I was heading northbound on Broadway, and I had a green light at 45th Street. As I was going through the intersection,
another car ran the red light from 45th Street and hit my passenger side.
- role: assistant
content: That must have been frightening. Was anyone else in your vehicle?
- role: user
content: No, I was alone, thankfully.
- role: assistant
content: And the other vehicle - did you get their information?
- role: user
content: Yes, it was a blue 2021 Toyota Camry. The driver's name was Robert Chen.
- role: assistant
content: Did you get his insurance information?
- role: user
content: Yes, he has Nationwide insurance. His policy number is... let me see... NW-5543298.
- role: assistant
content: That's very helpful. Did Mr. Chen admit fault at the scene?
- role: user
content: Well, he didn't exactly admit fault, but he did say he didn't see the light change. The police officer said it
was pretty clear he ran the red light.
- role: assistant
content: I see. Were there any witnesses?
- role: user
content: Yes! A woman named Maria Thompson saw everything. She gave her statement to the police. Her number is 916-555-3847.
- role: assistant
content: That's excellent. Having a witness will certainly help your claim. Now, let's talk about the damage to your vehicle.
Can you describe it?
- role: user
content: The entire passenger side is damaged. Both doors are dented in, the mirror is gone, and there's damage to the
front and rear quarter panels on that side.
- role: assistant
content: Is the vehicle driveable?
- role: user
content: No, they had to tow it. It's at Jerry's Auto Body on Riverside Drive.
- role: assistant
content: Do you have the address for Jerry's Auto Body?
- role: user
content: It's 3421 Riverside Drive, Sacramento, 95822.
- role: assistant
content: Thank you. We'll send an adjuster there. Do you have their phone number?
- role: user
content: Yes, it's 916-555-7823.
- role: assistant
content: Perfect. Now, let me review your coverage. You have comprehensive and collision with a $500 deductible, correct?
- role: user
content: I think so. I also have rental car coverage, right?
- role: assistant
content: Yes, you have rental reimbursement up to $40 per day for up to 30 days. Would you like me to arrange a rental
car for you?
- role: user
content: Yes, please. I need something to get to work.
- role: assistant
content: Of course. We work with Enterprise Rent-A-Car. There's a location at 2100 Fulton Avenue. Would that work for
you?
- role: user
content: Actually, is there anything closer to downtown? That's where I work.
- role: assistant
content: Let me check... Yes, there's an Enterprise at 1800 L Street, which is right downtown. Would that be better?
- role: user
content: Much better, thank you.
- role: assistant
content: I'll set that up for you. They'll have a midsize sedan ready for you tomorrow morning at 9 AM. Will that work?
- role: user
content: Can I get something today? I have to pick up my kids from school.
- role: assistant
content: Let me call them now and see what's available... Okay, they can have a car ready for you by 4 PM today. Will
that work?
- role: user
content: Perfect, my sister can get the kids and I'll pick up the rental after.
- role: assistant
content: Great. Your rental confirmation number is ENT-7739482. Just bring your driver's license and our claim number,
which is CLM-2024-894327.
- role: user
content: Let me write that down... CLM-2024-894327?
- role: assistant
content: Correct. Now, did you have any personal belongings damaged in the accident?
- role: user
content: Oh yes! My laptop was in the passenger seat and it got damaged. It's completely broken.
- role: assistant
content: What kind of laptop was it?
- role: user
content: A MacBook Pro, 14-inch, I bought it last year for about $2,200.
- role: assistant
content: Do you have the receipt or proof of purchase?
- role: user
content: I should have the email receipt from Apple. I can find it.
- role: assistant
content: Perfect. Please email that to claims@safeguardauto.com with your claim number in the subject line. Your policy
includes personal property coverage up to $2,500.
- role: user
content: Oh good, that will cover it. What about my medical check-up? The paramedics said I should see my doctor just
to be safe.
- role: assistant
content: Yes, your policy includes medical payments coverage up to $5,000. Just submit any medical bills related to this
accident and we'll cover them.
- role: user
content: How do I submit those?
- role: assistant
content: You can upload them through our mobile app, email them to the same claims address, or mail them to our office
at P.O. Box 5000, Sacramento, CA 95814.
- role: user
content: I'll probably use the app. Oh, what about missing work? I'm pretty sore and might need to take tomorrow off.
- role: assistant
content: Unfortunately, lost wages aren't covered under your current policy unless there's a serious injury requiring
extended time off. But definitely prioritize your health.
- role: user
content: Okay, understood. So what happens next?
- role: assistant
content: Our adjuster, Patricia Wong, will inspect your vehicle at Jerry's Auto Body within 24-48 hours. She'll call you
at the number we have on file - is 916-555-2234 still correct?
- role: user
content: Actually, I have a new number. It's 916-555-8899.
- role: assistant
content: I've updated that. Patricia will call you at 916-555-8899. She'll assess the damage and determine if it's repairable
or if it's a total loss.
- role: user
content: What happens if it's totaled?
- role: assistant
content: If it's deemed a total loss, we'll pay you the actual cash value of your vehicle minus your deductible. Based
on your 2019 Honda Accord with 45,000 miles, the estimated value would be around $22,000-24,000.
- role: user
content: I still owe about $8,000 on my car loan.
- role: assistant
content: That would be paid directly to your lender, Honda Financial Services, correct?
- role: user
content: Yes, that's right.
- role: assistant
content: We'd pay them first, and any remaining amount would come to you. You do have gap insurance on your policy, so
if there's any shortfall, that would be covered.
- role: user
content: Oh good, I forgot I had that.
- role: assistant
content: Now, since the other driver appears to be at fault based on the police report, we'll also pursue subrogation
against his insurance.
- role: user
content: What does that mean for me?
- role: assistant
content: It means we'll work to recover your deductible from his insurance company. If successful, you'll get your $500
deductible refunded.
- role: user
content: How long does that usually take?
- role: assistant
content: Subrogation can take 3-6 months typically, but we'll handle all of that for you. You don't need to do anything.
- role: user
content: Okay. Is there anything else I need to do right now?
- role: assistant
content: Just a few more things. First, don't discuss the accident with the other driver's insurance if they call. Refer
them to us.
- role: user
content: Got it. What if they've already called? Someone from Nationwide left a voicemail.
- role: assistant
content: Don't return their call. Let us handle all communication with them. Second, take photos of your injuries if you
have any bruising or cuts, even minor ones.
- role: user
content: I have some bruising on my shoulder from the seatbelt.
- role: assistant
content: Definitely photograph that, and date the photos. Email those to us as well. Also, keep all receipts for any expenses
related to this accident.
- role: user
content: Like what kind of expenses?
- role: assistant
content: Uber rides, medical co-pays, prescription medications, even mileage to and from medical appointments. We can
reimburse up to $500 in transportation expenses.
- role: user
content: I didn't know that. That's helpful.
- role: assistant
content: Is there anything else about the accident I should know? Any traffic cameras or business security cameras that
might have caught it?
- role: user
content: Actually, yes! There's a Wells Fargo on that corner that probably has cameras facing the intersection.
- role: assistant
content: Excellent. We'll request that footage. The police probably will too. Now, you should receive an email shortly
with all this information and your claim number.
- role: user
content: Perfect. Oh, one more thing - my registration and insurance cards were in the car. Can I get replacements?
- role: assistant
content: Of course. I'll email you temporary proof of insurance right now, and we'll mail new cards to your address at
892 Oak Lane, Sacramento, 95821. Is that address still correct?
- role: user
content: Yes, that's right.
- role: assistant
content: Perfect. They'll arrive in 5-7 business days. For your registration, you'll need to contact the DMV, but we can
provide a letter confirming your coverage if needed.
- role: user
content: Okay, I'll deal with the DMV later.
- role: assistant
content: I know this is a stressful situation, Ms. Martinez. We'll do everything we can to resolve this quickly. You'll
receive a call from Patricia Wong within 48 hours, and your rental car is confirmed for 4 PM today at the downtown Enterprise
location.
- role: user
content: Thank you so much, James. You've been really helpful.
- role: assistant
content: You're very welcome. Your claim number again is CLM-2024-894327. If you have any questions, call our claims line
anytime at 1-800-555-SAFE. Take care of yourself, and don't hesitate to seek medical attention if you feel worse.
user_question: I need to follow up on my accident claim. What was my claim number and when is the adjuster supposed to call
me?
evaluation_criteria: |
The agent should accurately retrieve and provide the claim number CLM-2024-894327 from the insurance claim conversation, along with the critical information that adjuster Patricia Wong will be calling within 24-48 hours. The agent should demonstrate clear recall of both the exact claim number and the specific timeframe for the adjuster's follow-up call. Ideally, the agent should also mention that Patricia Wong will be calling at the updated phone number 916-555-8899, showing attention to the detail that the phone number was changed during the conversation. The response should not confuse the claim number with other reference numbers mentioned in the conversation such as the police report number or the rental car confirmation number.
expected_behavior: The agent should accurately provide the claim number CLM-2024-894327 and mention that adjuster Patricia
Wong will call within 24-48 hours at the updated phone number
chapter3/user-memory-evaluation/test_cases/layer1/03_medical_appointment.yaml
+281
View File
@@ -0,0 +1,281 @@
test_id: layer1_03_medical_appointment
category: layer1
title: Healthcare Provider - Medical History and Appointment Scheduling
description: Test agent's ability to recall patient medical information and appointment details from comprehensive intake
call
conversation_histories:
- conversation_id: medical_intake_001
timestamp: '2024-11-12 09:00:00'
metadata:
business: Mountain View Medical Center
department: Patient Services
call_duration: 43 minutes
messages:
- role: user
content: Hi, I'm a new patient and need to schedule an appointment with a primary care doctor.
- role: assistant
content: Good morning! Welcome to Mountain View Medical Center. I'm Rachel from patient services. I'll help you get registered
and scheduled. First, may I have your full name?
- role: user
content: It's David Christopher Thompson.
- role: assistant
content: Thank you, Mr. Thompson. And your date of birth?
- role: user
content: August 9th, 1979.
- role: assistant
content: Perfect. What's your current address?
- role: user
content: 4521 Pine Valley Road, Denver, Colorado, 80203.
- role: assistant
content: And the best phone number to reach you?
- role: user
content: My cell is 303-555-7621.
- role: assistant
content: Do you have an email address for appointment reminders and patient portal access?
- role: user
content: Yes, it's dthompson79@email.com
- role: assistant
content: Excellent. Now, what type of insurance do you have?
- role: user
content: I have Blue Cross Blue Shield through my employer.
- role: assistant
content: Great. Can you provide your member ID number?
- role: user
content: It's BCB447829563.
- role: assistant
content: And the group number?
- role: user
content: Group number is 78845-001.
- role: assistant
content: Perfect. Who is your employer?
- role: user
content: I work for Meridian Software Solutions.
- role: assistant
content: Thank you. Now, are you the primary insurance holder?
- role: user
content: Yes, I am.
- role: assistant
content: Do you have any secondary insurance?
- role: user
content: No, just the one.
- role: assistant
content: Alright. Now I need to gather some medical history. Who was your previous primary care physician?
- role: user
content: Dr. Sarah Mitchell at Denver General. I've been seeing her for about five years but she's retiring.
- role: assistant
content: I see. We can request your records from Dr. Mitchell's office. Now, what brings you in for this appointment?
Any specific concerns?
- role: user
content: Well, I need a new primary doctor obviously, but I also have high blood pressure that needs monitoring, and I've
been having some back pain lately.
- role: assistant
content: I understand. How long have you had high blood pressure?
- role: user
content: It was diagnosed about three years ago. I'm on medication for it.
- role: assistant
content: What medication are you taking?
- role: user
content: Lisinopril, 10 milligrams once daily.
- role: assistant
content: Any other medications?
- role: user
content: I take atorvastatin for cholesterol, 20 milligrams at night, and I just started taking omeprazole for acid reflux
about a month ago.
- role: assistant
content: What dosage of omeprazole?
- role: user
content: 20 milligrams, once in the morning.
- role: assistant
content: Thank you. Do you take any vitamins or supplements?
- role: user
content: Just a daily multivitamin and vitamin D, 2000 units.
- role: assistant
content: Good. Any allergies to medications?
- role: user
content: Yes, I'm allergic to penicillin. I break out in hives.
- role: assistant
content: That's important to note. Any other allergies, including food or environmental?
- role: user
content: I have seasonal allergies, mainly tree pollen in the spring. And I'm lactose intolerant.
- role: assistant
content: Understood. Have you had any surgeries?
- role: user
content: I had my appendix removed in 2010 and knee surgery - ACL repair on my left knee in 2018.
- role: assistant
content: Any complications with those surgeries?
- role: user
content: No, both went smoothly.
- role: assistant
content: Good. Any hospitalizations besides those surgeries?
- role: user
content: I was hospitalized for pneumonia in 2021, just for two days.
- role: assistant
content: I see. Now, let me ask about family medical history. Are your parents living?
- role: user
content: My mother is, she's 72. My father passed away five years ago from a heart attack. He was 68.
- role: assistant
content: I'm sorry for your loss. Did your father have any other health conditions?
- role: user
content: He had diabetes and high blood pressure too.
- role: assistant
content: And your mother's health?
- role: user
content: She has arthritis and high cholesterol, but she's doing okay.
- role: assistant
content: Any siblings?
- role: user
content: One sister, she's 45 and healthy. Oh, but she was just diagnosed with thyroid issues.
- role: assistant
content: Any history of cancer in the family?
- role: user
content: My maternal grandmother had breast cancer, and my paternal grandfather had prostate cancer.
- role: assistant
content: Thank you for that information. Now, do you smoke or use tobacco products?
- role: user
content: No, never have.
- role: assistant
content: How about alcohol consumption?
- role: user
content: Maybe a glass of wine or two on weekends. Nothing excessive.
- role: assistant
content: Any recreational drug use?
- role: user
content: No, none.
- role: assistant
content: Good. How would you describe your exercise habits?
- role: user
content: I try to go to the gym three times a week, mostly cardio and some light weights. Though the back pain has limited
me lately.
- role: assistant
content: How long have you been experiencing the back pain?
- role: user
content: About six weeks now. It's lower back, gets worse when I sit for long periods.
- role: assistant
content: On a scale of 1 to 10, how would you rate the pain?
- role: user
content: Usually around a 4 or 5, but sometimes it spikes to a 7.
- role: assistant
content: Have you tried anything for the pain?
- role: user
content: Just ibuprofen and heating pads. They help a little.
- role: assistant
content: I see. Now, when was your last physical exam?
- role: user
content: About 14 months ago with Dr. Mitchell.
- role: assistant
content: And your last blood work?
- role: user
content: Same time, during that physical.
- role: assistant
content: When was your last eye exam?
- role: user
content: Oh, it's been a while. Maybe two years ago? My vision is fine though.
- role: assistant
content: And dental?
- role: user
content: I just went to the dentist last month actually.
- role: assistant
content: Good. Have you had any recent weight changes?
- role: user
content: I've gained about 10 pounds over the past year. I'm at 195 now.
- role: assistant
content: And your height?
- role: user
content: 5 feet 11 inches.
- role: assistant
content: Any issues with sleep?
- role: user
content: Sometimes I have trouble falling asleep, especially when my back is bothering me.
- role: assistant
content: Do you snore or has anyone told you that you stop breathing during sleep?
- role: user
content: My wife says I snore sometimes, but no breathing issues that she's noticed.
- role: assistant
content: Speaking of which, for emergency contact, would that be your wife?
- role: user
content: Yes, her name is Emily Thompson. Same address, and her cell is 303-555-7622.
- role: assistant
content: Perfect. Now, have you had your COVID vaccinations?
- role: user
content: Yes, fully vaccinated with two boosters. Last one was in September.
- role: assistant
content: Flu shot?
- role: user
content: Got that in October.
- role: assistant
content: Great. Any other immunizations we should know about?
- role: user
content: I got a tetanus booster about two years ago after I stepped on a nail.
- role: assistant
content: Good to know. Now, let's look at scheduling. We have Dr. Robert Chen or Dr. Lisa Wang available for primary care.
Do you have a preference for a male or female doctor?
- role: user
content: No preference really. Who has the earlier availability?
- role: assistant
content: Dr. Chen can see you next Thursday, November 21st at 2:30 PM, or Dr. Wang has an opening on Monday, November
25th at 10:00 AM.
- role: user
content: Thursday the 21st works better for me.
- role: assistant
content: Perfect. I'll schedule you with Dr. Chen for Thursday, November 21st at 2:30 PM. Since you're a new patient,
please arrive 15 minutes early to complete any remaining paperwork.
- role: user
content: Where exactly is the office?
- role: assistant
content: We're in the Medical Arts Building, 750 Broadway, Suite 200, Denver. There's a parking garage attached, and we
validate parking.
- role: user
content: Is that the building near the Denver Library?
- role: assistant
content: Yes, exactly! We're just two blocks south of the main library.
- role: user
content: Great, I know where that is. What should I bring?
- role: assistant
content: Please bring your insurance card, photo ID, a list of your current medications with dosages, and any medical
records you have from Dr. Mitchell.
- role: user
content: Should I fast for blood work?
- role: assistant
content: Since it's been over a year since your last labs, Dr. Chen will likely order blood work. It's best to fast for
12 hours just in case he wants a fasting glucose and lipid panel.
- role: user
content: Okay, so nothing after 2:30 AM that day?
- role: assistant
content: Correct, nothing except water after 2:30 AM. Also, take your morning medications as usual with water.
- role: user
content: Should I still take the blood pressure medication that morning?
- role: assistant
content: Yes, definitely continue your lisinopril. Never skip blood pressure medication.
- role: user
content: Got it. Will this be a full physical?
- role: assistant
content: Since you're a new patient and it's been over a year, Dr. Chen will likely do a comprehensive exam. Plan for
about 45 minutes.
- role: user
content: Perfect. Oh, do you have my prescription history? I'll need refills soon.
- role: assistant
content: Dr. Chen can handle refills during your visit. How many days of medication do you have left?
- role: user
content: About two weeks' worth for everything.
- role: assistant
content: That's perfect timing then. Dr. Chen will review everything and set up your refills. Your confirmation number
for this appointment is MV-789234.
- role: user
content: Let me write that down... MV-789234?
- role: assistant
content: Correct. You'll receive a reminder call the day before and a text message if you'd like.
- role: user
content: Text would be great.
- role: assistant
content: I've set that up for your cell phone. Is there anything else you need to know about your appointment?
- role: user
content: I think that covers everything. Thank you so much for being so thorough.
- role: assistant
content: You're very welcome, Mr. Thompson. We'll see you Thursday, November 21st at 2:30 PM with Dr. Chen. Remember to
fast 12 hours before and arrive 15 minutes early. Have a great day!
user_question: I need to confirm my upcoming doctor's appointment. What day and time was it scheduled for, and which doctor
am I seeing?
evaluation_criteria: |
The agent should accurately retrieve and provide the appointment scheduled for Thursday, November 21st at 2:30 PM with Dr. Robert Chen. The response must include both the correct date and time as well as the physician's name, demonstrating clear recall from the medical intake conversation. The agent should ideally also mention the confirmation number MV-789234 to show thorough memory retention. Additional helpful details might include the reminder to arrive 15 minutes early as a new patient or the 12-hour fasting requirement for potential blood work. The agent must not confuse Dr. Chen with Dr. Wang who was also mentioned as an option but not selected.
expected_behavior: The agent should clearly state the appointment is on Thursday, November 21st at 2:30 PM with Dr. Robert
Chen, possibly mentioning the confirmation number MV-789234
chapter3/user-memory-evaluation/test_cases/layer1/04_airline_booking.yaml
+254
View File
@@ -0,0 +1,254 @@
test_id: layer1_04_airline_booking
category: layer1
title: Airline Reservation - Flight Details and Passenger Information
description: Test agent's ability to recall specific flight booking details, seat assignments, and passenger preferences from
detailed reservation call
conversation_histories:
- conversation_id: airline_booking_001
timestamp: '2024-11-10 16:30:00'
metadata:
business: Pacific Airways
department: Reservations
call_duration: 48 minutes
messages:
- role: user
content: Hi, I need to book a round-trip flight from Seattle to Miami for a business conference.
- role: assistant
content: Good afternoon! Thank you for choosing Pacific Airways. I'm Marcus, and I'll help you with your reservation today.
When are you looking to travel?
- role: user
content: I need to fly out on December 5th and return on December 9th.
- role: assistant
content: Let me search for available flights. Do you have a preferred time of day for departure?
- role: user
content: I'd prefer a morning flight on the 5th, and afternoon or evening for the return.
- role: assistant
content: Perfect. And will this be for one passenger?
- role: user
content: Yes, just myself.
- role: assistant
content: May I have your full name as it appears on your ID?
- role: user
content: It's Rebecca Anne Sullivan.
- role: assistant
content: Thank you, Ms. Sullivan. I have several options for December 5th. We have flight PA 234 departing Seattle at
7:15 AM, arriving Miami at 3:45 PM with one stop in Dallas.
- role: user
content: How long is the layover in Dallas?
- role: assistant
content: It's a 1 hour and 30-minute layover.
- role: user
content: That's cutting it a bit close. What else do you have?
- role: assistant
content: There's PA 567 leaving at 9:30 AM, arriving at 6:20 PM with a stop in Phoenix. The layover is 2 hours and 15
minutes.
- role: user
content: That sounds better. What about non-stop flights?
- role: assistant
content: We have one non-stop, PA 890, departing at 10:45 AM and arriving at 7:10 PM. It's our premium route, so it's
about $200 more.
- role: user
content: What's the price difference exactly?
- role: assistant
content: The flight with the Phoenix connection is $487 in economy, and the non-stop is $685 in economy.
- role: user
content: And what about business class?
- role: assistant
content: For the non-stop, business class is $1,450. For the Phoenix connection, it's $980.
- role: user
content: Let's go with economy on the non-stop. I value my time more than the extra cost.
- role: assistant
content: Excellent choice. That's PA 890 on December 5th, departing Seattle at 10:45 AM. Now for your return on December
9th?
- role: user
content: What are the afternoon or evening options?
- role: assistant
content: I have PA 891, non-stop, departing Miami at 2:35 PM, arriving Seattle at 5:55 PM.
- role: user
content: That's perfect actually.
- role: assistant
content: Great! That's also $685 for economy. Would you like to select your seats now?
- role: user
content: Yes, please. I prefer an aisle seat.
- role: assistant
content: For the outbound flight, I have aisle seats available in rows 12, 18, 24, and 31. Do you have a preference?
- role: user
content: I'd like to be closer to the front. Row 12 sounds good.
- role: assistant
content: Row 12, seat C is available. It's an aisle seat on the right side of the aircraft. Would that work?
- role: user
content: Perfect.
- role: assistant
content: And for the return flight?
- role: user
content: Something similar if possible.
- role: assistant
content: I can put you in row 14, seat C, also an aisle on the right side.
- role: user
content: That works great.
- role: assistant
content: Wonderful. Now, are you a member of our Pacific Rewards program?
- role: user
content: No, I'm not.
- role: assistant
content: Would you like to join? It's free and you'll earn miles on this trip.
- role: user
content: Sure, why not.
- role: assistant
content: Excellent. I'll need your date of birth for the membership.
- role: user
content: January 18th, 1982.
- role: assistant
content: And your email address?
- role: user
content: rsullivan82@techmail.com
- role: assistant
content: Phone number?
- role: user
content: 206-555-4412.
- role: assistant
content: And your mailing address?
- role: user
content: 8923 Evergreen Way, Apartment 14B, Seattle, Washington, 98101.
- role: assistant
content: Perfect. Your Pacific Rewards number is PR-4478829. You'll earn about 5,400 miles from this round trip.
- role: user
content: Great. What about baggage? What's included?
- role: assistant
content: With your economy ticket, you get one personal item and one carry-on included. Checked bags are $35 for the first
bag, $45 for the second.
- role: user
content: I'll probably need to check one bag.
- role: assistant
content: Would you like to pre-pay for it now? It saves you $5 per bag.
- role: user
content: So it would be $30 instead of $35?
- role: assistant
content: Correct, $30 each way if you pre-pay now, so $60 total for round trip.
- role: user
content: Yes, let's do that.
- role: assistant
content: Added. Now, do you have any dietary restrictions for the in-flight meal service?
- role: user
content: I'm vegetarian.
- role: assistant
content: I've noted that. We'll have a vegetarian option available for you. Do you have TSA PreCheck or Global Entry?
- role: user
content: Yes, I have TSA PreCheck. The number is 842756931.
- role: assistant
content: I've added that to your reservation. It will appear on your boarding pass. Now, how would you like to pay for
this reservation?
- role: user
content: I'll use my Visa card.
- role: assistant
content: Ready when you are.
- role: user
content: The number is 4532-8876-9901-3345.
- role: assistant
content: And the expiration date?
- role: user
content: 10/26.
- role: assistant
content: The CVV code?
- role: user
content: '338.'
- role: assistant
content: And the billing zip code?
- role: user
content: 98101, same as my address.
- role: assistant
content: Is the name on the card Rebecca A Sullivan?
- role: user
content: Yes, that's correct.
- role: assistant
content: Perfect. Your total is $1,430 - that's two flights at $685 each plus $60 for checked baggage. Processing now...
- role: user
content: Sounds right.
- role: assistant
content: Your payment has been approved. Your confirmation number is PAC-778K4M.
- role: user
content: Let me get that... PAC-778K4M?
- role: assistant
content: That's correct. I'm emailing your itinerary to rsullivan82@techmail.com right now.
- role: user
content: Perfect. Oh, what about seat upgrades? Is there extra legroom available?
- role: assistant
content: Yes, we have Comfort Plus seats with 4 extra inches of legroom. Would you like to upgrade?
- role: user
content: How much is it?
- role: assistant
content: It's $89 per flight segment, so $178 for your round trip.
- role: user
content: I'll pass for now. Can I upgrade later if I change my mind?
- role: assistant
content: Absolutely. You can upgrade anytime before check-in through our website or mobile app.
- role: user
content: Good to know. What time should I arrive at the airport?
- role: assistant
content: For domestic flights, we recommend arriving 2 hours before departure. So by 8:45 AM for your December 5th flight.
- role: user
content: And I can check in online?
- role: assistant
content: Yes, online check-in opens 24 hours before departure. You can use our mobile app or website.
- role: user
content: Can I get my boarding pass on my phone?
- role: assistant
content: Yes, mobile boarding passes are available. You can also add them to your Apple Wallet or Google Pay.
- role: user
content: Perfect. What terminal does Pacific Airways use at SeaTac?
- role: assistant
content: We operate from Terminal B at Seattle-Tacoma International.
- role: user
content: And in Miami?
- role: assistant
content: We're in Terminal D at Miami International.
- role: user
content: What's your cancellation policy?
- role: assistant
content: Economy tickets are non-refundable, but you can change your flight for a $200 change fee plus any fare difference.
- role: user
content: What if I need to cancel for medical reasons?
- role: assistant
content: With a doctor's note, we can provide a credit for future travel valid for one year, minus a $100 processing fee.
- role: user
content: Hopefully I won't need that. What about travel insurance?
- role: assistant
content: We partner with TravelGuard. Would you like information about coverage?
- role: user
content: What does it cost?
- role: assistant
content: For your trip, comprehensive coverage would be $89. It covers trip cancellation, medical emergencies, lost baggage,
and delays.
- role: user
content: I'll think about it. Can I add it later?
- role: assistant
content: You have up to 14 days from today to add it and still get pre-existing condition coverage.
- role: user
content: Okay, I'll decide soon. Oh, one more thing - can I use the lounge in Seattle?
- role: assistant
content: Pacific Lounge access requires business class or Gold status in our rewards program. Day passes are available
for $59.
- role: user
content: Maybe on the way back. Is there anything else I should know?
- role: assistant
content: Just remember to check in 24 hours before each flight. Your record locator for all airport services is PAC-778K4M.
Both flights are on Boeing 737-900 aircraft with WiFi available for purchase.
- role: user
content: How much is the WiFi?
- role: assistant
content: It's $8 for one hour, $15 for the full flight on domestic routes.
- role: user
content: Good to know. I think that covers everything.
- role: assistant
content: 'Wonderful! To summarize: Rebecca Sullivan flying Pacific Airways on December 5th, PA 890 from Seattle at 10:45
AM to Miami, seat 12C. Returning December 9th, PA 891 at 2:35 PM, seat 14C. Confirmation PAC-778K4M. Total charged:
$1,430. You''re all set!'
- role: user
content: Perfect, thank you so much Marcus!
- role: assistant
content: My pleasure, Ms. Sullivan. Have a wonderful trip to Miami, and thank you for choosing Pacific Airways!
user_question: Can you remind me of my flight confirmation number and what seats I selected? I need to add it to my calendar.
evaluation_criteria: |
The agent should accurately retrieve and provide the confirmation number PAC-778K4M along with both seat assignments from the flight booking conversation - seat 12C for the outbound flight and seat 14C for the return flight. The response must demonstrate precise recall of both the confirmation code and the specific seats selected for each flight. The agent may also helpfully mention that both are aisle seats as discussed during booking, or reference the specific flight numbers (PA 890 outbound and PA 891 return). The agent must not confuse the confirmation number with other codes mentioned like the rewards number or mix up the seat assignments between the two flights.
expected_behavior: The agent should provide confirmation number PAC-778K4M and specify seat 12C for the outbound flight and
seat 14C for the return flight
chapter3/user-memory-evaluation/test_cases/layer1/05_internet_service_setup.yaml
+235
View File
@@ -0,0 +1,235 @@
test_id: layer1_05_internet_service
category: layer1
title: Internet and Cable Service Installation - Account Setup and Service Details
description: Test agent's ability to recall service package details, installation appointment, and account information from
comprehensive service setup call
conversation_histories:
- conversation_id: internet_setup_001
timestamp: '2024-11-08 11:00:00'
metadata:
business: Velocity Communications
department: New Service Sales
call_duration: 45 minutes
messages:
- role: user
content: Hi, I just moved to a new house and need to set up internet and cable service.
- role: assistant
content: Welcome to Velocity Communications! I'm Derek, and I'll be happy to help you set up your services today. First,
let me get your service address.
- role: user
content: It's 7622 Willow Creek Drive, Austin, Texas, 78759.
- role: assistant
content: Great location! Let me check what services are available there... Excellent news! You qualify for our fiber optic
services with speeds up to 2 gigabits. May I have your name?
- role: user
content: I'm Kevin Zhang.
- role: assistant
content: Thank you, Mr. Zhang. Are you looking for internet only, or would you like to bundle with cable TV and phone?
- role: user
content: I definitely need internet. Tell me about the cable packages too.
- role: assistant
content: 'Sure! For internet, we have three tiers: Essential at 300 Mbps for $49.99, Performance at 1 Gig for $79.99,
and Ultimate at 2 Gigs for $119.99 per month.'
- role: user
content: I work from home and do a lot of video calls. What would you recommend?
- role: assistant
content: For working from home with video calls, I'd recommend at least the Performance 1 Gig plan. It provides excellent
upload and download speeds for smooth video conferencing.
- role: user
content: What about the cable packages?
- role: assistant
content: We have Select with 125+ channels for $64.99, Preferred with 200+ channels plus HBO and Showtime for $94.99,
and Premier with 300+ channels plus all premium channels for $134.99.
- role: user
content: Do you have sports channels in the Select package?
- role: assistant
content: Select includes ESPN, ESPN2, and Fox Sports regional networks. For NFL Network, NBA TV, and NFL RedZone, you'd
need Preferred or higher.
- role: user
content: I'm a big basketball fan, so I'd need NBA TV. What's the total for Performance internet and Preferred cable?
- role: assistant
content: Performance internet at $79.99 plus Preferred cable at $94.99 would be $174.98, but I can offer you a bundle
discount of $25 per month, bringing it to $149.98.
- role: user
content: That's not bad. What about equipment fees?
- role: assistant
content: The fiber modem/router combo is $12 per month. Each cable box is $7.99 for standard or $12.99 for DVR-enabled
boxes. How many TVs do you have?
- role: user
content: I have three TVs - living room, master bedroom, and my home office.
- role: assistant
content: I'd recommend a DVR box for your living room so you can record games and shows, and standard boxes for the other
two. That would be $12.99 plus two at $7.99 each.
- role: user
content: So that's about $29 in equipment fees?
- role: assistant
content: Actually, $28.97 for the cable boxes plus $12 for the modem/router, so $40.97 total in equipment fees.
- role: user
content: What would my total monthly bill be then?
- role: assistant
content: Your services at $149.98 plus equipment at $40.97 equals $190.95 per month, plus taxes and fees which are typically
around $12-15 in your area.
- role: user
content: Are there any promotions for new customers?
- role: assistant
content: Yes! New customers get the first three months at 50% off the service charges, and we waive the installation fee
which is normally $99.99.
- role: user
content: So I'd pay about $95 for the first three months?
- role: assistant
content: For the services portion, yes - $74.99 for the first three months, plus the $40.97 equipment fees, so roughly
$116 before taxes.
- role: user
content: Okay, let's do it. When can you install?
- role: assistant
content: Let me check our installation calendar... I have openings this Saturday November 11th in the afternoon, or next
Tuesday November 14th in the morning.
- role: user
content: What time on Tuesday?
- role: assistant
content: We have an 8 AM to 12 PM window or a 10 AM to 2 PM window available.
- role: user
content: I'll take the 10 AM to 2 PM on Tuesday the 14th.
- role: assistant
content: Perfect! I've scheduled your installation for Tuesday, November 14th between 10 AM and 2 PM. Our technician will
call 30 minutes before arrival.
- role: user
content: Do I need to be home?
- role: assistant
content: Yes, someone 18 or older needs to be present as the technician will need access inside to set up your equipment
and activate services.
- role: user
content: That's fine. Now what information do you need from me?
- role: assistant
content: I'll need your date of birth for the account setup.
- role: user
content: April 3rd, 1987.
- role: assistant
content: And your Social Security number for the credit check?
- role: user
content: It's 455-78-9234.
- role: assistant
content: Thank you. Running that now... You're approved with no deposit required! What's the best phone number to reach
you?
- role: user
content: My cell is 512-555-8823.
- role: assistant
content: Email address for your account and bills?
- role: user
content: kzhang87@email.com
- role: assistant
content: Perfect. Would you like paperless billing?
- role: user
content: Yes, definitely paperless.
- role: assistant
content: Great choice! You'll get a $5 monthly discount for paperless billing with autopay. Would you like to set up autopay?
- role: user
content: What are the autopay options?
- role: assistant
content: You can use a checking account, savings account, or credit/debit card. The payment processes on your due date
each month.
- role: user
content: I'll use my checking account. It's with Chase Bank.
- role: assistant
content: I'll need your routing number and account number.
- role: user
content: Routing is 111000614 and account is 8829447651.
- role: assistant
content: Thank you. Your autopay is set up and your adjusted monthly rate with the paperless discount is $185.95 plus
taxes and fees.
- role: user
content: When will my first bill be?
- role: assistant
content: Your first bill will generate about 10 days after installation, so around November 24th, due 20 days later on
December 14th.
- role: user
content: And that will be at the promotional rate?
- role: assistant
content: Yes, your first three bills will show the 50% discount on services. After that, it goes to regular pricing.
- role: user
content: What channels are included in HBO?
- role: assistant
content: You get HBO, HBO2, HBO Family, HBO Latino, HBO Signature, HBO Comedy, HBO Zone, plus HBO Max streaming access.
- role: user
content: Oh, streaming is included?
- role: assistant
content: Yes! With Preferred package, you get HBO Max and Showtime streaming apps included at no extra charge. You can
use them on any device.
- role: user
content: That's great. What about a streaming device for my TVs?
- role: assistant
content: Our cable boxes have built-in streaming apps, or you can use your own devices like Roku or Apple TV with your
login credentials.
- role: user
content: Do the cable boxes need to be connected to the internet?
- role: assistant
content: They work best when connected to your home network for on-demand content and streaming apps. The technician will
connect them during installation.
- role: user
content: How many devices can use the internet simultaneously?
- role: assistant
content: With the 1 Gig plan, you can easily support 25+ devices simultaneously. Our router supports up to 150 connected
devices.
- role: user
content: Is the WiFi dual-band?
- role: assistant
content: Yes, it's WiFi 6 with both 2.4GHz and 5GHz bands, plus it automatically optimizes which band each device uses.
- role: user
content: Can I use my own router?
- role: assistant
content: You can, but you'd still need our modem for the fiber connection. If you use your own router, we can reduce the
equipment fee to $7 for just the modem.
- role: user
content: I'll stick with yours for now. Is there a contract?
- role: assistant
content: No long-term contract required! It's month-to-month service. You can cancel anytime with 30 days notice.
- role: user
content: What if I want to upgrade or downgrade services?
- role: assistant
content: You can change your package anytime. Upgrades take effect immediately, downgrades at the start of your next billing
cycle.
- role: user
content: Good to know. What's my account number going to be?
- role: assistant
content: Your new account number is VEL-89923476. You'll use this for all service requests.
- role: user
content: VEL-89923476, got it. Do you have a mobile app?
- role: assistant
content: Yes! The Velocity app lets you manage your account, pay bills, troubleshoot issues, and even watch live TV on
your mobile devices.
- role: user
content: Can I set up parental controls?
- role: assistant
content: Absolutely. Both the internet router and cable boxes have comprehensive parental controls you can manage through
the app or web portal.
- role: user
content: Perfect. What if I have technical issues?
- role: assistant
content: We have 24/7 technical support at 1-800-VELOCITY. Most issues can be resolved remotely, but we offer free service
calls if needed.
- role: user
content: Is there anything I need to do before the installation?
- role: assistant
content: Just ensure clear access to where you want the equipment installed and have your TVs ready. The technician will
handle all the wiring and setup.
- role: user
content: How long does installation usually take?
- role: assistant
content: For three TVs and internet, typically 2-3 hours depending on your home's wiring situation.
- role: user
content: Sounds good. I think I'm all set.
- role: assistant
content: 'Perfect! Let me confirm everything: Kevin Zhang at 7622 Willow Creek Drive, Austin. Installation on Tuesday,
November 14th, 10 AM-2 PM. Performance 1 Gig internet and Preferred cable package with one DVR and two standard boxes.
Account number VEL-89923476. Monthly rate $185.95 after discounts, with first three months at 50% off services. Is everything
correct?'
- role: user
content: Yes, that's all correct. Thank you!
- role: assistant
content: You're welcome, Mr. Zhang! You'll receive a confirmation email at kzhang87@email.com shortly. Our technician
will see you on Tuesday. Thank you for choosing Velocity Communications!
user_question: What's my Velocity account number? I need it to download the mobile app and set up my online account.
evaluation_criteria: |
The agent should accurately retrieve and provide the Velocity Communications account number VEL-89923476 that was assigned during the service setup call. The response must include the exact account number with the correct format including the VEL prefix and all eight digits. The agent should demonstrate clear recall of this specific identifier without confusing it with other numbers mentioned in the conversation such as the installation date, pricing, or address numbers. The response should be direct and precise since the user needs this information to set up their online account and mobile app access.
expected_behavior: The agent should directly provide the account number VEL-89923476 that was assigned during the service
setup
chapter3/user-memory-evaluation/test_cases/layer1/06_credit_card_application.yaml
+233
View File
@@ -0,0 +1,233 @@
test_id: layer1_06_credit_card_app
category: layer1
title: Credit Card Application - Financial Information and Card Details
description: Test agent's ability to recall credit card terms, application details, and financial information from detailed
application call
conversation_histories:
- conversation_id: credit_card_app_001
timestamp: '2024-11-06 13:20:00'
metadata:
business: Premier Bank Credit Services
department: Card Applications
call_duration: 41 minutes
messages:
- role: user
content: Hi, I received a pre-approved offer for your Rewards Plus credit card and I'd like to apply.
- role: assistant
content: Excellent! Thank you for your interest in the Premier Bank Rewards Plus card. I'm Sandra, and I'll help you with
your application today. May I have the invitation code from your offer?
- role: user
content: Yes, it's RWD-447829-Q3.
- role: assistant
content: Perfect! I see you're pre-approved for up to $15,000 credit limit. Let me verify some information. Is this Patricia
Chen?
- role: user
content: Yes, that's me.
- role: assistant
content: Wonderful, Ms. Chen. Can you confirm your date of birth?
- role: user
content: June 14th, 1990.
- role: assistant
content: Thank you. And your current address?
- role: user
content: 2156 Mission Street, Apartment 8C, San Francisco, California, 94110.
- role: assistant
content: How long have you been at this address?
- role: user
content: About three years now.
- role: assistant
content: Excellent. Now, let me tell you about the Rewards Plus card benefits. You'll earn 3% cash back on dining and
entertainment, 2% on groceries and gas, and 1% on everything else.
- role: user
content: Is there an annual fee?
- role: assistant
content: No annual fee at all. Plus, you get a $200 bonus after spending $1,000 in the first three months.
- role: user
content: That sounds great. What's the interest rate?
- role: assistant
content: Based on your pre-approval, you qualify for our best rate of 16.99% APR for purchases. Balance transfers are
at 14.99% for the first 12 months, then the regular rate.
- role: user
content: Do you offer a 0% introductory rate?
- role: assistant
content: Yes! You get 0% APR on purchases for the first 15 months, then it goes to the 16.99% rate.
- role: user
content: That's really good. What about balance transfer fees?
- role: assistant
content: Balance transfers have a 3% fee, with a minimum of $10. But with the lower rate for 12 months, it often saves
money overall.
- role: user
content: I have about $3,000 I'd like to transfer from another card.
- role: assistant
content: That's perfectly fine. We can set that up once your card is approved. Now, I need to verify your employment information.
Where do you work?
- role: user
content: I work at Bay Area Tech Solutions as a senior project manager.
- role: assistant
content: How long have you been with Bay Area Tech Solutions?
- role: user
content: Five and a half years.
- role: assistant
content: Excellent stability. What's your annual income?
- role: user
content: My base salary is $135,000, and I usually get about $20,000 in bonuses.
- role: assistant
content: So total annual income around $155,000?
- role: user
content: Yes, that's right.
- role: assistant
content: Do you have any additional sources of income?
- role: user
content: I do some freelance consulting, maybe $10,000 a year.
- role: assistant
content: I'll note that as $165,000 total annual income. Now, do you rent or own your home?
- role: user
content: I rent.
- role: assistant
content: What's your monthly rent payment?
- role: user
content: $2,850 per month.
- role: assistant
content: Do you have any other credit cards currently?
- role: user
content: Yes, I have two - a Chase Sapphire and an American Express Blue.
- role: assistant
content: What are the credit limits on those?
- role: user
content: Chase is $12,000 and Amex is $8,000.
- role: assistant
content: And roughly what are your current balances?
- role: user
content: The Chase has about $3,000 which I want to transfer, and the Amex has maybe $500.
- role: assistant
content: Thank you. Any other loans - auto, student loans, personal loans?
- role: user
content: I have a car loan with about $15,000 remaining.
- role: assistant
content: What's your monthly payment on that?
- role: user
content: $425 per month.
- role: assistant
content: Perfect. Now, what's your mother's maiden name for security purposes?
- role: user
content: It's Williams.
- role: assistant
content: Thank you. Let me process your application now... Congratulations! You're approved for a $14,000 credit limit.
- role: user
content: That's fantastic! When will I receive the card?
- role: assistant
content: Your card will arrive in 7-10 business days via USPS to your Mission Street address.
- role: user
content: Can I expedite it?
- role: assistant
content: Absolutely! For $25, we can send it FedEx for delivery in 2-3 business days.
- role: user
content: Let's do that. I have some purchases coming up.
- role: assistant
content: Perfect. It will arrive by November 9th. Now, would you like to set up that balance transfer?
- role: user
content: Yes, the $3,000 from my Chase card.
- role: assistant
content: I'll need the account number for your Chase card.
- role: user
content: It's 4147-2298-8834-9002.
- role: assistant
content: And which bank issues that card?
- role: user
content: Chase Bank.
- role: assistant
content: Perfect. The $3,000 transfer will process within 7-14 days after you receive and activate your card. The 3% fee
of $90 will be added to your first statement.
- role: user
content: That works. Can I add an authorized user?
- role: assistant
content: Yes, you can add authorized users at no additional cost. Would you like to add someone now?
- role: user
content: Yes, my husband, James Chen.
- role: assistant
content: I'll need his date of birth.
- role: user
content: March 22nd, 1988.
- role: assistant
content: His card will arrive with yours. He'll earn rewards on his purchases that go into your account.
- role: user
content: Great. How do I track my rewards?
- role: assistant
content: You can see your cash back balance on your monthly statement, online account, or mobile app. Rewards post after
each statement closes.
- role: user
content: When can I redeem them?
- role: assistant
content: You can redeem anytime once you have at least $25 in rewards. Options include statement credit, direct deposit,
or check.
- role: user
content: Do the rewards expire?
- role: assistant
content: No, they never expire as long as your account remains open and in good standing.
- role: user
content: What about foreign transaction fees?
- role: assistant
content: The Rewards Plus card has no foreign transaction fees, making it perfect for international travel.
- role: user
content: That's great, I travel to Asia twice a year. What about rental car insurance?
- role: assistant
content: Yes, you get automatic collision damage waiver when you rent with your card. Plus extended warranty on purchases
and purchase protection.
- role: user
content: Is there fraud protection?
- role: assistant
content: Absolutely. Zero liability for unauthorized charges, and we monitor 24/7 for suspicious activity. You can also
set up instant alerts.
- role: user
content: How do I set up online access?
- role: assistant
content: Once you receive your card, go to premier-bank.com/activate. You'll create a username and password there.
- role: user
content: Can I choose my payment due date?
- role: assistant
content: Yes, after your first statement, you can call or go online to change your due date to any day of the month you
prefer.
- role: user
content: I'd like it around the 15th of each month.
- role: assistant
content: You can set that up after your first statement generates. Your temporary due date will be the 5th of each month.
- role: user
content: What's the minimum payment?
- role: assistant
content: The minimum is 2% of your balance or $25, whichever is greater.
- role: user
content: And the credit limit again was?
- role: assistant
content: You're approved for $14,000.
- role: user
content: Can I request an increase later?
- role: assistant
content: Yes, after six months of on-time payments, you can request an increase online or by phone.
- role: user
content: Is there a cash advance option?
- role: assistant
content: Yes, cash advances are available up to 30% of your credit limit, so up to $4,200 for you. The fee is 5% or $10
minimum, and the APR is 24.99%.
- role: user
content: I'll avoid that. What's the customer service number?
- role: assistant
content: It's 1-800-555-CARD, available 24/7. Your account number for reference is 5524-8897-2234-4001.
- role: user
content: Let me write that down... 5524-8897-2234-4001?
- role: assistant
content: That's correct. That's your new card number. You'll create a PIN when you activate the card.
- role: user
content: Perfect. Is there anything else I should know?
- role: assistant
content: Just remember to activate your card as soon as it arrives to enable your purchase protection benefits and to
start the balance transfer. You'll need the three-digit activation code that comes with the card.
- role: user
content: Sounds good. Thank you so much for your help!
- role: assistant
content: You're very welcome, Ms. Chen! Your Premier Bank Rewards Plus card with a $14,000 limit will arrive via FedEx
by November 9th. Remember, you have 0% APR for 15 months and will earn that $200 bonus after $1,000 in purchases. Thank
you for choosing Premier Bank!
user_question: What's my new credit card number? I want to add it to my password manager for when I start online shopping.
evaluation_criteria: |
The agent should accurately retrieve and provide the complete credit card number 5524-8897-2234-4001 that was assigned during the application process. The response must include all 16 digits in the correct sequence, maintaining the proper formatting either with dashes as shown or as a continuous string of numbers. The agent should demonstrate precise recall of this specific card number and may helpfully mention that this is for the Premier Bank Rewards Plus card with the $14,000 credit limit. The agent must not confuse this card number with other numbers mentioned in the conversation such as phone numbers, SSN, or the credit limit amount.
expected_behavior: The agent should provide the credit card number 5524-8897-2234-4001 that was assigned during the application
chapter3/user-memory-evaluation/test_cases/layer1/07_car_rental.yaml
+224
View File
@@ -0,0 +1,224 @@
test_id: layer1_07_car_rental
category: layer1
title: Car Rental for Business Trip - Reservation Details
description: Test agent's ability to recall car rental confirmation and vehicle details from comprehensive booking call
conversation_histories:
- conversation_id: car_rental_001
timestamp: '2024-11-14 10:15:00'
metadata:
business: Enterprise Rent-A-Car
department: Reservations
call_duration: 44 minutes
messages:
- role: user
content: Hello, I need to rent a car for a business trip to Phoenix next week.
- role: assistant
content: Good morning! Thank you for calling Enterprise. I'm Brian, and I'll help you with your rental today. When will
you need the vehicle?
- role: user
content: I'm flying in Tuesday morning, November 21st, and leaving Friday evening, November 24th.
- role: assistant
content: Perfect, so pickup on the 21st and return on the 24th. Will you be picking up at Phoenix Sky Harbor Airport?
- role: user
content: Yes, that's right.
- role: assistant
content: And returning to the same location?
- role: user
content: Yes, same airport.
- role: assistant
content: What time does your flight arrive on Tuesday?
- role: user
content: I land at 9:45 AM.
- role: assistant
content: So you'll probably be at our counter around 10:30 AM after getting your luggage?
- role: user
content: Yes, that sounds about right.
- role: assistant
content: And your return flight on Friday?
- role: user
content: My flight is at 6:30 PM, so I'd like to return the car around 4:30 PM.
- role: assistant
content: That works perfectly. What size vehicle are you looking for?
- role: user
content: I'll be doing a lot of driving to client sites. Something comfortable, maybe midsize or full-size.
- role: assistant
content: I have several options. Our midsize category includes cars like the Toyota Camry or Nissan Altima at $52 per
day.
- role: user
content: What about full-size?
- role: assistant
content: Full-size would be something like a Chevrolet Malibu or Chrysler 300 at $58 per day.
- role: user
content: Not much difference in price. What about gas mileage?
- role: assistant
content: The midsize typically gets about 32 MPG combined, the full-size around 28 MPG.
- role: user
content: I'll probably drive 500 miles total. The midsize should be fine.
- role: assistant
content: Good choice. Now, are you an Enterprise Plus member?
- role: user
content: No, I don't think so.
- role: assistant
content: Would you like to join? It's free and you earn points toward free rental days.
- role: user
content: Sure, why not.
- role: assistant
content: Great! I'll need your full name as it appears on your driver's license.
- role: user
content: It's Gregory Alan Mitchell.
- role: assistant
content: Thank you, Mr. Mitchell. Date of birth?
- role: user
content: May 12th, 1981.
- role: assistant
content: Email address for your account?
- role: user
content: gmitchell81@businessmail.com
- role: assistant
content: Phone number?
- role: user
content: 602-555-7734.
- role: assistant
content: Current home address?
- role: user
content: 5521 North Central Avenue, Phoenix, Arizona, 85012.
- role: assistant
content: Oh, you're local to Phoenix?
- role: user
content: No, sorry, I'm getting confused. I live in Dallas. 7823 Preston Road, Dallas, Texas, 75252.
- role: assistant
content: No problem! Dallas address noted. Your Enterprise Plus number is EP-9982734. You'll earn about 400 points on
this rental.
- role: user
content: How many points do I need for a free day?
- role: assistant
content: You need 6,000 points for a free day, so this gets you started.
- role: user
content: Okay, good to know. Now, about insurance - what do you offer?
- role: assistant
content: We have several coverage options. The basic damage waiver is $29 per day and covers the vehicle if it's damaged
or stolen.
- role: user
content: My company credit card includes rental car insurance. I'll skip it.
- role: assistant
content: That's fine. Just make sure you're using that card for the rental. What about personal accident insurance?
- role: user
content: No, I have good health insurance.
- role: assistant
content: Understood. Would you like to add roadside assistance for $6.99 per day?
- role: user
content: Is that really necessary?
- role: assistant
content: It covers flat tires, lockouts, jump starts, and towing. In an unfamiliar city, some people find it worthwhile.
- role: user
content: You know what, add it. Better safe than sorry.
- role: assistant
content: Added. Now, would you like to prepay for gas? You can return it at any fuel level for $3.89 per gallon.
- role: user
content: What's gas running in Phoenix right now?
- role: assistant
content: Around $3.50 to $3.70 per gallon currently.
- role: user
content: So your prepay is actually more expensive. I'll just fill it up myself.
- role: assistant
content: That's usually the more economical choice. Would you like GPS navigation for $13.99 per day?
- role: user
content: No, I'll use my phone.
- role: assistant
content: Alright. Do you need any special equipment like a car seat or ski rack?
- role: user
content: No, nothing like that.
- role: assistant
content: Perfect. Let me calculate your total. Four days at $52 per day is $208, plus roadside assistance at $6.99 per
day is $27.96.
- role: user
content: So about $236 before taxes?
- role: assistant
content: Exactly $235.96 before taxes and fees. With Arizona taxes and airport fees, your estimated total is $287.42.
- role: user
content: That's within my company's daily allowance. Let's book it.
- role: assistant
content: Excellent. I'll need a credit card to hold the reservation.
- role: user
content: I'll use my corporate AmEx. It's 3782-445589-31004.
- role: assistant
content: And the expiration date?
- role: user
content: March 2026.
- role: assistant
content: Name on the card?
- role: user
content: It says Gregory Mitchell, TechCorp Inc.
- role: assistant
content: Perfect. Your reservation is confirmed. Your confirmation number is E7739482M.
- role: user
content: Let me write that down... E7739482M?
- role: assistant
content: That's correct. E as in Enterprise, 7739482M as in Mitchell.
- role: user
content: Got it. Where exactly do I go when I land?
- role: assistant
content: Follow signs to the Rental Car Center. You'll take the PHX Sky Train from your terminal - it's free and runs
every 3-5 minutes.
- role: user
content: How long does that take?
- role: assistant
content: About 5-7 minutes. The Rental Car Center is the last stop, you can't miss it.
- role: user
content: And Enterprise is in that building?
- role: assistant
content: Yes, we're on the first floor. With Enterprise Plus, you can skip the counter and go directly to the Plus aisle
to choose your car.
- role: user
content: Oh, that's convenient! Any car in that aisle?
- role: assistant
content: Any car in the Midsize section of the Plus aisle. Your name will be on the board with your aisle number.
- role: user
content: What if I want to upgrade when I'm there?
- role: assistant
content: You can choose from a higher category if available. You'd just pay the difference at the exit booth.
- role: user
content: Good to know. What documents do I need?
- role: assistant
content: Just your driver's license and the credit card you're using for payment.
- role: user
content: The same AmEx I just gave you?
- role: assistant
content: Yes, that's perfect. The card needs to match the driver's name.
- role: user
content: It does. Oh, one more thing - I might have a colleague join me. Can I add a second driver?
- role: assistant
content: Yes, additional drivers are $15 per day in Arizona. They'll need to show their license at pickup.
- role: user
content: Actually, she's also a TechCorp employee. Does that matter?
- role: assistant
content: Let me check if TechCorp has a corporate account with us... Yes! TechCorp employees get free additional drivers.
Just mention the corporate code TC-8847 at pickup.
- role: user
content: Oh excellent! I didn't know we had that.
- role: assistant
content: Yes, and actually that code also gives you a 10% discount. Let me apply that to your reservation... Your new
total is $258.68.
- role: user
content: Even better! This is great.
- role: assistant
content: I'm glad I could help save you some money. Is there anything else you need?
- role: user
content: Can I get a receipt emailed to me?
- role: assistant
content: Yes, we'll email your final receipt to gmitchell81@businessmail.com when you return the car.
- role: user
content: Perfect. I think that's everything.
- role: assistant
content: 'Wonderful! Just to confirm: Gregory Mitchell, Enterprise Plus member EP-9982734, picking up a midsize car at
Phoenix Sky Harbor on November 21st around 10:30 AM, returning November 24th at 4:30 PM. Confirmation E7739482M, total
$258.68 with corporate discount. You''re all set!'
- role: user
content: That's all correct. Thanks so much, Brian!
- role: assistant
content: You're very welcome, Mr. Mitchell. Have a great trip to Phoenix!
user_question: What's my Enterprise confirmation number and what corporate discount code should I mention? I want to make
sure I get the discount.
evaluation_criteria: |
The agent should accurately retrieve and provide both the rental confirmation number E7739482M and the TechCorp corporate discount code TC-8847 from the car rental conversation. The response must include the exact confirmation number with all characters in the correct sequence, as well as the corporate code that provides the 10% discount. The agent should demonstrate clear recall of both pieces of information that are essential for the user to receive the negotiated discount rate. The agent may also mention that the corporate code is associated with TechCorp or reference the 10% discount it provides. The response must not confuse these codes with other identifiers mentioned such as the Enterprise Plus member number EP-9982734.
expected_behavior: The agent should provide confirmation E7739482M and corporate code TC-8847 for the TechCorp discount
chapter3/user-memory-evaluation/test_cases/layer1/08_hotel_reservation.yaml
+202
View File
@@ -0,0 +1,202 @@
test_id: layer1_08_hotel_reservation
category: layer1
title: Hotel Booking for Anniversary - Reservation and Package Details
description: Test agent's ability to recall hotel confirmation, room upgrade, and special package details
conversation_histories:
- conversation_id: hotel_booking_001
timestamp: '2024-11-09 19:30:00'
metadata:
business: Hilton Hotels
department: Reservations
call_duration: 46 minutes
messages:
- role: user
content: Hi, I want to book a special weekend for our 10th anniversary in San Diego.
- role: assistant
content: Good evening! Congratulations on your upcoming anniversary! I'm Maria from Hilton Reservations. I'd love to help
make it special. Which dates were you considering?
- role: user
content: We're looking at the weekend of December 8th and 9th.
- role: assistant
content: That's a Saturday and Sunday. Were you planning to check in Friday night or Saturday?
- role: user
content: Friday night, December 8th, and check out Sunday the 10th.
- role: assistant
content: Perfect, two nights. Which Hilton property in San Diego interests you? We have several options.
- role: user
content: We were thinking something on the water, maybe the Bayfront?
- role: assistant
content: Excellent choice! The Hilton San Diego Bayfront is our premier waterfront property. It's right on the bay with
beautiful views.
- role: user
content: That sounds perfect. What room options do you have?
- role: assistant
content: For your anniversary, might I suggest our bay view rooms or even a suite? We have standard bay view rooms starting
at $379 per night.
- role: user
content: What about suites?
- role: assistant
content: We have Junior Suites with bay views at $529 per night, or our Premium Suites at $729 per night with panoramic
views and a separate living area.
- role: user
content: The Junior Suite sounds nice. Tell me more about it.
- role: assistant
content: The Junior Suite is 650 square feet with a king bed, separate seating area, floor-to-ceiling windows overlooking
the bay, and a marble bathroom with soaking tub.
- role: user
content: That sounds romantic. Do you have any anniversary packages?
- role: assistant
content: Yes! Our Romance Package includes chocolate-covered strawberries, champagne on arrival, late checkout, and breakfast
in bed for $125 additional.
- role: user
content: Oh, that's nice. What time is the late checkout?
- role: assistant
content: With the package, you can check out as late as 2 PM instead of the standard 11 AM.
- role: user
content: Let's add that package. So that would be the Junior Suite with the Romance Package?
- role: assistant
content: Yes, exactly. Let me calculate that for you. Two nights in the Junior Suite at $529 per night is $1,058, plus
the $125 Romance Package.
- role: user
content: What about taxes?
- role: assistant
content: San Diego hotel tax is 12.7%, so that adds about $150 to your total.
- role: user
content: So roughly $1,330 total?
- role: assistant
content: Yes, $1,333 to be exact. Now, may I have your name to create the reservation?
- role: user
content: It's Richard and Susan Thompson.
- role: assistant
content: Thank you, Mr. Thompson. And a contact phone number?
- role: user
content: 619-555-9821.
- role: assistant
content: Email address for confirmation?
- role: user
content: rthompson@techcompany.com
- role: assistant
content: Perfect. Are you a Hilton Honors member?
- role: user
content: My wife is. Her number is... let me find it... HH294857301.
- role: assistant
content: Excellent! I'll attach this to her account so you'll earn points. That's about 13,000 points for this stay.
- role: user
content: Wow, that's a lot of points.
- role: assistant
content: Yes, and with her Silver status, she gets a 20% bonus, so actually about 15,600 points.
- role: user
content: She'll be happy about that. Can we request a specific floor?
- role: assistant
content: Of course! The Junior Suites are on floors 15 through 20. Do you have a preference?
- role: user
content: Higher is better for us. Maybe 18 or above?
- role: assistant
content: I'll note a request for floor 18 or higher. I actually have room 1912 available - 19th floor, corner unit with
exceptional views.
- role: user
content: Oh, a corner room? That sounds perfect!
- role: assistant
content: Yes, it's one of our best Junior Suites. Extra windows on two sides. Shall I reserve that specific room?
- role: user
content: Yes, please do that.
- role: assistant
content: Room 1912 is now reserved for you. Would you like to make any dinner reservations?
- role: user
content: Does the hotel have restaurants?
- role: assistant
content: Yes, we have three. Vela is our signature waterfront restaurant, Hudson & Nash is more casual, and SWIM is our
poolside bar and grill.
- role: user
content: Is Vela good for anniversary dinner?
- role: assistant
content: Absolutely! It's perfect for special occasions. They have a special anniversary dessert if you mention it's your
celebration.
- role: user
content: Can you make us a reservation for Saturday night?
- role: assistant
content: I'd be happy to. What time would you prefer?
- role: user
content: Maybe 7:30 PM?
- role: assistant
content: Let me check... Yes, I can confirm 7:30 PM on Saturday, December 9th for two at Vela. I'll note it's your anniversary.
- role: user
content: Wonderful. What about parking?
- role: assistant
content: Valet parking is $55 per night, or self-parking is $42 per night.
- role: user
content: We'll probably Uber mostly, but good to know.
- role: assistant
content: Smart choice. The hotel is walking distance to the Gaslamp Quarter and Seaport Village too.
- role: user
content: Oh that's great for exploring. Is there a spa?
- role: assistant
content: Yes, our spa offers couples massages which are very popular for anniversaries. Would you like information about
booking?
- role: user
content: How much are those?
- role: assistant
content: The 50-minute couples massage is $580, or the 80-minute is $750. They book up quickly for weekends.
- role: user
content: Maybe we'll book that when we arrive. What about the pool?
- role: assistant
content: We have four pools including an adults-only infinity pool on the fourth floor with bay views.
- role: user
content: That sounds nice. Is it heated?
- role: assistant
content: Yes, all our pools are heated year-round. The infinity pool is especially beautiful at sunset.
- role: user
content: Perfect. Now, for payment - what do you need?
- role: assistant
content: I'll need a credit card to guarantee the reservation. The full payment is due at check-in.
- role: user
content: I'll use my Visa. It's 4556-7890-1234-5678.
- role: assistant
content: And the expiration date?
- role: user
content: 07/26.
- role: assistant
content: The name on the card?
- role: user
content: Richard P Thompson.
- role: assistant
content: Thank you. Your reservation is confirmed! Your confirmation number is HSD-445782.
- role: user
content: Let me write that down... HSD-445782?
- role: assistant
content: That's correct. You'll receive an email confirmation shortly to rthompson@techcompany.com.
- role: user
content: What's your cancellation policy?
- role: assistant
content: You can cancel up to 48 hours before arrival without penalty. Within 48 hours, there's a one-night charge.
- role: user
content: Good to know, though we definitely won't be canceling!
- role: assistant
content: I'm sure you'll have a wonderful anniversary! Is there anything special you'd like in the room? Rose petals,
specific flowers?
- role: user
content: Could you arrange for roses?
- role: assistant
content: Of course! A dozen red roses will be in your room upon arrival, compliments of the hotel for your anniversary.
- role: user
content: That's so thoughtful, thank you!
- role: assistant
content: Our pleasure! I'll also notify the front desk that you're celebrating so they can ensure everything is perfect.
- role: user
content: I really appreciate all your help, Maria.
- role: assistant
content: 'It''s my pleasure, Mr. Thompson! To summarize: December 8-10, room 1912 Junior Suite with bay view on the 19th
floor, Romance Package included, dinner at Vela on Saturday at 7:30 PM, confirmation HSD-445782. Total is $1,333.'
- role: user
content: That's all perfect.
- role: assistant
content: Wonderful! Happy early anniversary to you and Susan. We look forward to making your celebration memorable!
- role: user
content: Thank you so much!
- role: assistant
content: You're very welcome! Have a lovely evening and we'll see you December 8th!
user_question: Can you remind me of my Hilton confirmation number and what room number we reserved? My wife is asking about
our anniversary trip details.
evaluation_criteria: |
The agent should accurately retrieve and provide both the Hilton confirmation number HSD-445782 and the specific room number 1912 that was reserved for the anniversary trip. The response must include the exact confirmation code and the correct room number, demonstrating clear recall of these reservation details. The agent may also helpfully mention that room 1912 is a corner Junior Suite on the 19th floor with bay views, as these details were emphasized during the booking conversation. Since the user mentions this is for their anniversary trip, the agent might also recall the Romance Package or other anniversary-related arrangements. The response must not confuse these numbers with other numerical information mentioned in the conversation such as dates, prices, or rewards numbers.
expected_behavior: The agent should provide confirmation HSD-445782 and specify room 1912 was reserved
chapter3/user-memory-evaluation/test_cases/layer1/09_home_security.yaml
+219
View File
@@ -0,0 +1,219 @@
test_id: layer1_09_home_security
category: layer1
title: Home Security System Installation - Service Agreement Details
description: Test agent's ability to recall security system package details and installation appointment
conversation_histories:
- conversation_id: security_install_001
timestamp: '2024-11-11 14:45:00'
metadata:
business: ADT Security Services
department: Sales and Installation
call_duration: 48 minutes
messages:
- role: user
content: Hi, I've been having some break-ins in my neighborhood and want to install a security system.
- role: assistant
content: I understand your concern, and you're making a smart decision. I'm Carlos from ADT Security Services. I'll help
you find the right protection for your home. First, may I have your name?
- role: user
content: I'm Nancy Rodriguez.
- role: assistant
content: Thank you, Ms. Rodriguez. What city are you in?
- role: user
content: I'm in Houston, Texas, in the Heights area.
- role: assistant
content: I know that area well. Tell me about your home - is it a single-family house, townhome, or apartment?
- role: user
content: It's a two-story house, about 2,400 square feet.
- role: assistant
content: How many entry doors do you have?
- role: user
content: Front door, back door, and a side door to the garage.
- role: assistant
content: So three entry points. How many windows on the ground floor?
- role: user
content: Let me think... six in the front, four on each side, and four in the back. So 18 total.
- role: assistant
content: That's quite a few windows to secure. Do you have a garage?
- role: user
content: Yes, a two-car attached garage.
- role: assistant
content: And do you have any sliding glass doors?
- role: user
content: Yes, one leading to the backyard from the living room.
- role: assistant
content: Good to know. Based on your home's layout, I'd recommend our Smart Home Security package. It includes door sensors
for all entry points, window sensors, motion detectors, and our smart hub.
- role: user
content: How many window sensors would I get?
- role: assistant
content: The package includes sensors for up to 15 windows. We'd prioritize the most vulnerable ones - typically windows
hidden from street view or near door locks.
- role: user
content: What about the other three windows?
- role: assistant
content: We can add additional sensors for $40 each, or use glass break detectors which cover multiple windows in a room
for $120 each.
- role: user
content: The glass break detectors sound more efficient.
- role: assistant
content: They are. One detector can cover a 25-foot radius. We'd probably need just one additional detector for full coverage.
- role: user
content: Okay, let's plan on that. What about cameras?
- role: assistant
content: Our package includes two outdoor cameras. Most people place them at the front door and back door. Would you like
additional cameras?
- role: user
content: I'd like one for the garage and maybe one for the side of the house.
- role: assistant
content: Smart thinking. Additional cameras are $199 each. So that would be two more cameras.
- role: user
content: Yes, let's do four cameras total. Do they record?
- role: assistant
content: Yes, they record to the cloud. The basic package includes 30 days of video storage. You can upgrade to 60 days
for $10 more per month.
- role: user
content: 30 days should be enough. What's the monthly monitoring fee?
- role: assistant
content: The Smart Home Security package with professional monitoring is $52.99 per month with a 36-month agreement.
- role: user
content: That's a three-year contract?
- role: assistant
content: Yes, but that locks in your rate. Without a contract, it's $62.99 monthly.
- role: user
content: I'll take the contract for the lower rate. What about installation?
- role: assistant
content: Professional installation is $299, but we're running a promotion this month where it's only $99.
- role: user
content: Oh, that's a good deal. When could you install?
- role: assistant
content: Let me check our Houston schedule... I have availability this Thursday, November 16th, or next Monday, November
20th.
- role: user
content: What time on Thursday?
- role: assistant
content: I have a morning slot from 8 AM to 12 PM, or afternoon from 1 PM to 5 PM.
- role: user
content: I'll take Thursday morning, 8 to noon.
- role: assistant
content: Perfect! I've scheduled you for Thursday, November 16th, between 8 AM and noon. The technician will call 30 minutes
before arrival.
- role: user
content: Do I need to be home?
- role: assistant
content: Yes, someone 18 or older needs to be present for the full installation, which typically takes 3-4 hours.
- role: user
content: That's fine, I'll work from home that day. Now, does the system work if the power goes out?
- role: assistant
content: Yes! The control panel has a backup battery that provides 24 hours of power. The cellular connection continues
working too.
- role: user
content: Good. How do I arm and disarm it?
- role: assistant
content: Multiple ways - a keypad by each door, a key fob remote, or our mobile app on your smartphone.
- role: user
content: How many key fobs do I get?
- role: assistant
content: The package includes two key fobs. Additional ones are $35 each.
- role: user
content: Two is enough. My husband and I will each have one. What about smoke detectors?
- role: user
content: Actually, wait - before smoke detectors, what's my account number going to be?
- role: assistant
content: Your ADT account number will be ADT-7789234. I'll also give you a service code for the installation.
- role: user
content: What's the service code?
- role: assistant
content: Your installation service code is SVC-44821. The technician will ask for this.
- role: user
content: Let me write both down... ADT-7789234 and SVC-44821?
- role: assistant
content: Exactly right. Now, you were asking about smoke detectors?
- role: user
content: Yes, are those included?
- role: assistant
content: The package includes one smoke detector. Additional monitored smoke detectors are $65 each. How many total would
you like?
- role: user
content: I should probably have one upstairs and one downstairs.
- role: assistant
content: That's wise. So one additional smoke detector for $65.
- role: user
content: Yes. Do you also monitor carbon monoxide?
- role: assistant
content: Yes, CO detectors are $100 each. With your gas appliances, I'd recommend at least one.
- role: user
content: Okay, add one of those too.
- role: assistant
content: Added. Let me calculate your total setup... Equipment package is $849, two additional cameras at $199 each, one
glass break detector at $120, one smoke detector at $65, one CO detector at $100, installation at $99.
- role: user
content: What's the total?
- role: assistant
content: Your one-time equipment and installation cost is $1,631, and monthly monitoring is $52.99.
- role: user
content: Can I finance the equipment?
- role: assistant
content: Yes! We can spread it over 12 months at 0% interest, which would be about $136 per month plus your monitoring.
- role: user
content: So about $190 total per month for the first year?
- role: assistant
content: Yes, $188.91 to be exact for the first 12 months, then just $52.99 after that.
- role: user
content: That's manageable. Let's do the financing.
- role: assistant
content: Excellent. I'll need some information for the financing approval. What's your social security number?
- role: user
content: 523-84-7791.
- role: assistant
content: Date of birth?
- role: user
content: October 15, 1978.
- role: assistant
content: Current address?
- role: user
content: 1847 Heights Boulevard, Houston, Texas, 77008.
- role: assistant
content: How long have you lived there?
- role: user
content: About six years.
- role: assistant
content: Employment status?
- role: user
content: I'm employed full-time as a marketing director.
- role: assistant
content: Excellent. Let me run this through... Congratulations! You're approved for the 0% financing.
- role: user
content: Great! What are the next steps?
- role: assistant
content: I'll email your contract to review and sign electronically. What's your email?
- role: user
content: nrodriguez78@email.com
- role: assistant
content: Perfect. You'll receive that within the hour. Once signed, we'll ship your equipment for the Thursday installation.
- role: user
content: Will the installer explain everything?
- role: assistant
content: Yes, they'll train you on the entire system, set up your app, and make sure you're comfortable before leaving.
- role: user
content: What if I have problems after installation?
- role: assistant
content: We have 24/7 customer support. Plus, your account includes two free service calls per year if needed.
- role: user
content: That's reassuring. Oh, does this integrate with smart home devices?
- role: assistant
content: Yes! It works with Alexa, Google Assistant, and most smart locks and lights.
- role: user
content: Perfect, I have Alexa. Anything else I should know?
- role: assistant
content: Just remember your account number ADT-7789234 for any future service needs, and have your service code SVC-44821
ready for the installer Thursday morning.
- role: user
content: Got it. Thank you so much, Carlos!
- role: assistant
content: You're welcome, Ms. Rodriguez! You're all set for Thursday, November 16th, 8 AM to noon. Your home will be much
more secure very soon!
user_question: What's my ADT account number and installation service code? The technician just arrived and is asking for them.
evaluation_criteria: |
The agent should accurately retrieve and provide both the ADT account number ADT-7789234 and the installation service code SVC-44821 from the home security setup conversation. Since the technician has just arrived and is asking for these codes, the response must include both pieces of information exactly as provided during the call. The agent should demonstrate clear recall of both the account number (which will be needed for future service) and the service code (specifically required for the installation). The response must maintain the correct format and sequence for both codes and not confuse them with each other or with other numbers mentioned in the conversation such as the address, phone numbers, or pricing information.
expected_behavior: The agent should provide account number ADT-7789234 and service code SVC-44821
chapter3/user-memory-evaluation/test_cases/layer1/10_pharmacy_transfer.yaml
+217
View File
@@ -0,0 +1,217 @@
test_id: layer1_10_pharmacy_transfer
category: layer1
title: Pharmacy Prescription Transfer - Medication and Insurance Details
description: Test agent's ability to recall prescription details and insurance information from pharmacy call
conversation_histories:
- conversation_id: pharmacy_transfer_001
timestamp: '2024-11-13 10:00:00'
metadata:
business: Walgreens Pharmacy
department: Pharmacy Services
call_duration: 39 minutes
messages:
- role: user
content: Hello, I need to transfer my prescriptions from CVS to your pharmacy.
- role: assistant
content: Good morning! I'm Rachel from Walgreens pharmacy. I'll be happy to help you transfer your prescriptions. First,
may I have your name?
- role: user
content: It's Matthew Harrison.
- role: assistant
content: Thank you, Mr. Harrison. And your date of birth?
- role: user
content: September 5th, 1975.
- role: assistant
content: Phone number?
- role: user
content: 773-555-4429.
- role: assistant
content: Current address?
- role: user
content: 3924 North Lincoln Avenue, Chicago, Illinois, 60613.
- role: assistant
content: Do you have prescription insurance?
- role: user
content: Yes, I have UnitedHealthcare.
- role: assistant
content: Can I get your member ID number?
- role: user
content: It's UHC887234561.
- role: assistant
content: And the group number?
- role: user
content: Group number is 445KL8.
- role: assistant
content: Is the cardholder name Matthew Harrison?
- role: user
content: Yes, I'm the primary.
- role: assistant
content: Perfect. Now, which CVS location are your prescriptions at?
- role: user
content: The one on Broadway, near Belmont. I think it's 3637 North Broadway.
- role: assistant
content: Yes, I know that location. What prescriptions do you need transferred?
- role: user
content: I have four medications. Let me get my bottles... Okay, I have metoprolol for blood pressure.
- role: assistant
content: What strength and how often do you take it?
- role: user
content: 50 milligrams twice daily.
- role: assistant
content: And the prescription number from the CVS bottle?
- role: user
content: It's RX7739482.
- role: assistant
content: Got it. What else?
- role: user
content: Rosuvastatin for cholesterol, 20 milligrams once at night.
- role: assistant
content: Prescription number for that one?
- role: user
content: RX7739483.
- role: assistant
content: And the third medication?
- role: user
content: Jardiance for diabetes, 10 milligrams once in the morning.
- role: assistant
content: That's a newer diabetes medication. Prescription number?
- role: user
content: RX7739484.
- role: assistant
content: And the fourth one?
- role: user
content: Gabapentin 300 milligrams, three times daily for nerve pain.
- role: assistant
content: Prescription number?
- role: user
content: RX7739485.
- role: assistant
content: Who is your prescribing physician?
- role: user
content: Dr. Amanda Foster at Northwestern Medical Group.
- role: assistant
content: Do all four prescriptions have refills remaining?
- role: user
content: Yes, they all have at least 6 refills left.
- role: assistant
content: Excellent. When did you last fill these?
- role: user
content: About a week ago, so I have three weeks of medication left.
- role: assistant
content: Perfect timing. Do you have any drug allergies?
- role: user
content: I'm allergic to codeine - it makes me break out in hives.
- role: assistant
content: Important to note. Any other medications, vitamins, or supplements?
- role: user
content: I take vitamin D3, 2000 units daily, and fish oil.
- role: assistant
content: Good to know. Now let me check your insurance coverage for these medications...
- role: user
content: Will the copays be the same?
- role: assistant
content: Let me see... Your metoprolol is a Tier 1 generic, $10 copay. Rosuvastatin is also Tier 1, another $10.
- role: user
content: That's what I was paying at CVS.
- role: assistant
content: Good. The Jardiance is Tier 3, showing a $47 copay.
- role: user
content: Yes, that's right.
- role: assistant
content: And gabapentin is Tier 1, so $10.
- role: user
content: So $77 total for all four?
- role: assistant
content: Correct. Now, would you like us to synchronize these so they all refill on the same day?
- role: user
content: Oh, can you do that?
- role: assistant
content: Yes! We'll do short fills on some to align them all to the same refill date. Makes it much easier.
- role: user
content: That would be great. Just one trip to the pharmacy.
- role: assistant
content: Exactly. Would you like automatic refills?
- role: user
content: What does that mean?
- role: assistant
content: We'll automatically start preparing your refills 5 days before you run out and text you when they're ready.
- role: user
content: Yes, that sounds convenient.
- role: assistant
content: Perfect. Now, I'll contact CVS to transfer these. We'll need about 2-3 hours to process everything.
- role: user
content: So I can pick them up this afternoon?
- role: assistant
content: We'll have them ready by 2 PM today. Is text the best way to notify you?
- role: user
content: Yes, text is perfect.
- role: assistant
content: Great. Your Walgreens prescription account number will be WG-9847221.
- role: user
content: Should I write that down?
- role: assistant
content: Yes, you'll need it for refills or any pharmacy questions. WG-9847221.
- role: user
content: Got it. Which Walgreens should I go to?
- role: assistant
content: You can use any Walgreens, but I'm setting your home store as the one at 3419 North Broadway, just a few blocks
from the CVS you were using.
- role: user
content: Oh that's convenient, even closer to me.
- role: assistant
content: Yes, and we're open until 10 PM every day. We also have a drive-through.
- role: user
content: Perfect for bad weather days. Do you have an app?
- role: assistant
content: Yes, the Walgreens app lets you manage prescriptions, order refills, and even see pill images to identify medications.
- role: user
content: I'll download that. Can I pay through the app?
- role: assistant
content: You can pre-pay and just pick up at the counter or drive-through. Very quick.
- role: user
content: Great. Oh, does Dr. Foster need to approve the transfer?
- role: assistant
content: No, we handle the transfer between pharmacies directly. But I see here she's already in our system, so future
prescriptions from her will come straight to us.
- role: user
content: Good. What about vaccine records? I got my flu shot at CVS.
- role: assistant
content: We can request your immunization history from CVS as well. Would you like us to?
- role: user
content: Yes, please. I never remember when I got what vaccine.
- role: assistant
content: We'll add that to your profile. Speaking of vaccines, have you had your COVID booster this fall?
- role: user
content: Not yet, been meaning to.
- role: assistant
content: We have appointments available. You could get it when you pick up your prescriptions if you have time.
- role: user
content: Maybe next time. I have meetings this afternoon.
- role: assistant
content: No problem. Now, do you need any other pharmacy services? We do medication consultations, blood pressure checks,
diabetes supplies.
- role: user
content: I do need to order test strips for my glucose meter.
- role: assistant
content: What meter do you use?
- role: user
content: The OneTouch Verio.
- role: assistant
content: Those test strips are covered by your insurance. $35 for a box of 100. Should I add those to today's order?
- role: user
content: Yes, please do that.
- role: assistant
content: Added. So your total today will be $112 - the four medications plus test strips.
- role: user
content: Sounds right.
- role: assistant
content: Perfect. Everything will be ready at 2 PM. You'll get a text when it's ready. Your account number is WG-9847221,
and you're all set up for auto-refills with medication synchronization.
- role: user
content: This was easier than I expected. Thanks, Rachel!
- role: assistant
content: You're very welcome, Mr. Harrison! We'll take great care of you here at Walgreens. See you this afternoon!
user_question: What's my Walgreens prescription account number? I need it to set up the mobile app.
evaluation_criteria: |
The agent should accurately retrieve and provide the Walgreens prescription account number WG-9847221 that was assigned during the pharmacy transfer conversation. The response must include the exact account number with the correct WG prefix and all seven digits in the proper sequence. This account number is specifically needed for setting up the mobile app, so the agent should provide it clearly and accurately. The response should demonstrate precise recall of this identifier without confusing it with other numbers mentioned in the conversation such as CVS prescription numbers, copay amounts, or phone numbers.
expected_behavior: The agent should provide Walgreens account number WG-9847221
chapter3/user-memory-evaluation/test_cases/layer1/11_mortgage_application.yaml
+301
View File
@@ -0,0 +1,301 @@
test_id: layer1_11_mortgage_application
category: layer1
title: Mortgage Application - Financial Details Retrieval
description: Test agent's ability to recall detailed financial information provided during a complex mortgage application call
conversation_histories:
- conversation_id: mortgage_app_001
timestamp: "2024-11-20 09:15:00"
metadata:
business: "Wells Fargo Home Lending"
department: "Mortgage Applications"
call_duration: "63 minutes"
messages:
- role: user
content: "Hi, I'd like to apply for a home mortgage loan."
- role: assistant
content: "Good morning! I'm Robert from Wells Fargo Home Lending. I'll help you with your mortgage application today. Let's start with some basic information. What's your full legal name?"
- role: user
content: "David Alexander Thompson."
- role: assistant
content: "Thank you, Mr. Thompson. And your date of birth?"
- role: user
content: "July 22nd, 1982."
- role: assistant
content: "What's your Social Security number?"
- role: user
content: "652-41-8897."
- role: assistant
content: "And your current address?"
- role: user
content: "2156 Riverside Drive, Apartment 14C, Chicago, Illinois 60601."
- role: assistant
content: "How long have you lived at this address?"
- role: user
content: "About three years and four months."
- role: assistant
content: "Are you currently renting or do you own?"
- role: user
content: "I'm renting. The monthly rent is $2,450."
- role: assistant
content: "Who's your landlord or management company?"
- role: user
content: "Citywide Property Management LLC."
- role: assistant
content: "Thank you. Now, let's discuss your employment. Current employer?"
- role: user
content: "I work for McKenzie Financial Advisors."
- role: assistant
content: "Your position there?"
- role: user
content: "Senior Financial Analyst."
- role: assistant
content: "How long have you been with McKenzie Financial?"
- role: user
content: "Six years and two months."
- role: assistant
content: "And your annual salary?"
- role: user
content: "My base salary is $145,000 per year."
- role: assistant
content: "Do you receive any bonuses or additional compensation?"
- role: user
content: "Yes, I typically get an annual bonus. Last year it was $28,500."
- role: assistant
content: "Any other sources of income?"
- role: user
content: "I do some freelance consulting, averaging about $1,800 per month."
- role: assistant
content: "How long have you been doing the consulting work?"
- role: user
content: "For about two years now."
- role: assistant
content: "Do you have a business name or is it under your personal name?"
- role: user
content: "It's under Thompson Consulting LLC."
- role: assistant
content: "Great. Now let's talk about your assets. What's your total in checking accounts?"
- role: user
content: "I have about $18,500 in my Chase checking account."
- role: assistant
content: "Savings accounts?"
- role: user
content: "I have $67,000 in my Wells Fargo savings and another $45,000 in a Marcus by Goldman Sachs high-yield savings."
- role: assistant
content: "Any investment accounts?"
- role: user
content: "Yes, I have a Vanguard brokerage account with about $125,000 and a TD Ameritrade account with $38,000."
- role: assistant
content: "What about retirement accounts?"
- role: user
content: "My 401k with my employer has around $285,000, and I have a Roth IRA at Fidelity with $78,000."
- role: assistant
content: "Excellent. Now, any liabilities? Credit card debt?"
- role: user
content: "I have three credit cards. The Visa has a $2,300 balance, MasterCard is paid off, and my Amex has about $1,100."
- role: assistant
content: "What are the credit limits on those cards?"
- role: user
content: "The Visa is $15,000 limit, MasterCard is $12,000, and Amex is $25,000."
- role: assistant
content: "Any auto loans?"
- role: user
content: "Yes, I have a car loan with BMW Financial. The balance is $31,500."
- role: assistant
content: "Monthly payment?"
- role: user
content: "$685 per month."
- role: assistant
content: "Interest rate on that loan?"
- role: user
content: "It's 3.9%."
- role: assistant
content: "Any student loans?"
- role: user
content: "No, I paid those off two years ago."
- role: assistant
content: "Other debts or monthly obligations?"
- role: user
content: "No, that's everything."
- role: assistant
content: "Now, about the property you're looking to purchase. What's the address?"
- role: user
content: "It's 789 Oak Park Avenue, Evanston, Illinois 60201."
- role: assistant
content: "Purchase price?"
- role: user
content: "$625,000."
- role: assistant
content: "How much are you planning to put down?"
- role: user
content: "I'm planning to put down 20%, so $125,000."
- role: assistant
content: "So you're looking to borrow $500,000?"
- role: user
content: "Exactly."
- role: assistant
content: "Have you already made an offer?"
- role: user
content: "Yes, it was accepted. We're in attorney review until December 5th."
- role: assistant
content: "When's the planned closing date?"
- role: user
content: "January 15th, 2025."
- role: assistant
content: "Is this going to be your primary residence?"
- role: user
content: "Yes, absolutely."
- role: assistant
content: "Are you purchasing alone or with someone else?"
- role: user
content: "Just me, I'm buying it alone."
- role: assistant
content: "Are you a first-time homebuyer?"
- role: user
content: "No, I owned a condo about eight years ago but sold it when I relocated for work."
- role: assistant
content: "What did you sell that property for?"
- role: user
content: "I sold it for $285,000 back in 2016."
- role: assistant
content: "Did you have any gain or loss on that sale?"
- role: user
content: "I made about a $45,000 profit after all expenses."
- role: assistant
content: "Good. Now, do you have a preference for loan type? 30-year fixed, 15-year fixed, or adjustable rate?"
- role: user
content: "I'm interested in a 30-year fixed rate."
- role: assistant
content: "Based on your credit score range, which we'll verify, current rates are around 6.875% for a 30-year fixed with 20% down."
- role: user
content: "Speaking of credit score, what do you need?"
- role: assistant
content: "We'll pull your credit report. Do you know your approximate score?"
- role: user
content: "Last I checked it was 782."
- role: assistant
content: "Excellent score. That should qualify you for our best rates. Have you had any late payments in the last two years?"
- role: user
content: "No, no late payments at all."
- role: assistant
content: "Any bankruptcies or foreclosures?"
- role: user
content: "No, never."
- role: assistant
content: "Have you co-signed any loans for anyone?"
- role: user
content: "Actually yes, I co-signed my sister's student loan about five years ago."
- role: assistant
content: "What's the balance on that loan?"
- role: user
content: "It's down to about $18,000 now."
- role: assistant
content: "Is she current on payments?"
- role: user
content: "Yes, she's never missed a payment."
- role: assistant
content: "Good. Now, for the down payment, where will those funds come from?"
- role: user
content: "From my savings accounts - the Wells Fargo and Marcus accounts I mentioned."
- role: assistant
content: "Will any of the down payment be a gift from family?"
- role: user
content: "No, it's all my own savings."
- role: assistant
content: "Have you made any large deposits to your accounts in the last 60 days?"
- role: user
content: "Just my regular salary and the consulting income. Oh, and I got my annual bonus of $28,500 deposited about three weeks ago."
- role: assistant
content: "That's fine, we'll just need documentation for that. Now, regarding the property, what type of home is it?"
- role: user
content: "It's a single-family home, built in 1998."
- role: assistant
content: "How many bedrooms and bathrooms?"
- role: user
content: "Four bedrooms, three and a half bathrooms."
- role: assistant
content: "Square footage?"
- role: user
content: "About 3,200 square feet."
- role: assistant
content: "Is there a homeowners association?"
- role: user
content: "Yes, the HOA fee is $185 per month."
- role: assistant
content: "What does that cover?"
- role: user
content: "Landscaping of common areas, snow removal, and maintenance of the community park and pool."
- role: assistant
content: "Annual property taxes?"
- role: user
content: "They're about $14,800 per year."
- role: assistant
content: "Has a home inspection been done yet?"
- role: user
content: "Scheduled for next Tuesday, November 26th."
- role: assistant
content: "Good. Now, let me calculate your debt-to-income ratio. Your monthly gross income including the consulting would be about $14,267, correct?"
- role: user
content: "That sounds right."
- role: assistant
content: "With the mortgage payment, taxes, insurance, HOA, and your car payment, your total monthly housing and debt payments would be approximately $4,285. That gives you a debt-to-income ratio of about 30%, which is excellent."
- role: user
content: "That's good to hear."
- role: assistant
content: "For the loan application, I'll need several documents. Do you have your last two years of tax returns?"
- role: user
content: "Yes, I have 2022 and 2023 returns ready."
- role: assistant
content: "Last two months of pay stubs?"
- role: user
content: "I can download those from our HR portal."
- role: assistant
content: "Last two months of bank statements for all accounts?"
- role: user
content: "I'll need to gather those."
- role: assistant
content: "And we'll need the purchase agreement and your employer's contact information for verification."
- role: user
content: "The purchase agreement is with my attorney. My HR department number is 312-555-8900."
- role: assistant
content: "Perfect. Once we receive all documents, the underwriting process typically takes 3-4 weeks."
- role: user
content: "That should work with our January 15th closing."
- role: assistant
content: "Yes, we'll have plenty of time. I'll email you the full application and document checklist. What's your email address?"
- role: user
content: "It's dthompson82@email.com"
- role: assistant
content: "Great. I'll send that within the hour. Any questions about the mortgage process?"
- role: user
content: "What about mortgage insurance?"
- role: assistant
content: "With 20% down, you won't need PMI - private mortgage insurance."
- role: user
content: "Perfect. And when do I lock in the interest rate?"
- role: assistant
content: "You can lock anytime between now and closing. We offer 30, 45, or 60-day locks. Given your January 15th closing, I'd recommend a 60-day lock for safety."
- role: user
content: "What if rates drop after I lock?"
- role: assistant
content: "We have a one-time float-down option if rates drop by at least 0.25% from your locked rate."
- role: user
content: "That's good to know. I think I'll wait a week or two before locking."
- role: assistant
content: "That's fine. Just keep in mind rates can change daily. I'll monitor them and keep you informed."
- role: user
content: "Please do. I think that covers everything I needed to know today."
- role: assistant
content: "Excellent. I'll process your application and send you the document list. My direct line is 312-555-7600 if you have any questions."
- role: user
content: "Thank you, Robert. I'll gather those documents and send them over soon."
- role: assistant
content: "Perfect. Congratulations on your home purchase, Mr. Thompson. We'll make sure everything goes smoothly for your closing."
user_question: "What was my total liquid assets amount that I mentioned for the mortgage application?"
evaluation_criteria: |
The agent should correctly recall and calculate the total liquid assets mentioned:
- Chase checking: $18,500
- Wells Fargo savings: $67,000
- Marcus high-yield savings: $45,000
- Total liquid assets: $130,500
The agent should provide this specific total, not including investment or retirement accounts which are not typically considered immediately liquid for mortgage purposes.
chapter3/user-memory-evaluation/test_cases/layer1/12_gym_membership.yaml
+227
View File
@@ -0,0 +1,227 @@
test_id: layer1_12_gym_membership
category: layer1
title: Gym Membership Cancellation - Contract Details Retrieval
description: Test agent's ability to recall membership terms, fees, and personal training package details from a lengthy gym membership discussion
conversation_histories:
- conversation_id: gym_cancel_001
timestamp: "2024-11-18 16:30:00"
metadata:
business: "Lifetime Fitness"
department: "Member Services"
call_duration: "52 minutes"
messages:
- role: user
content: "Hi, I need to discuss my membership and possibly make some changes."
- role: assistant
content: "Hello! This is Marcus from Lifetime Fitness Member Services. I'd be happy to help you with your membership. May I have your member ID or phone number?"
- role: user
content: "My member ID is LF-2021-78432."
- role: assistant
content: "Thank you. Let me pull up your account... I see here Ms. Rachel Chen?"
- role: user
content: "Yes, that's me."
- role: assistant
content: "Perfect. I see you've been a member since March 2021. You currently have our Diamond membership. What changes were you considering?"
- role: user
content: "Well, I'm thinking about canceling, but I want to understand my options first."
- role: assistant
content: "I understand. Let me review your current membership details. You're paying $189 per month for Diamond, which includes all club access, group classes, and pool/spa facilities."
- role: user
content: "Right. And I also have that personal training package."
- role: assistant
content: "Yes, I see you purchased a 24-session package with trainer James Morrison for $2,400 back in September."
- role: user
content: "How many sessions do I have left?"
- role: assistant
content: "Let me check... You've completed 8 sessions, so you have 16 remaining."
- role: user
content: "And those expire when?"
- role: assistant
content: "Personal training packages are valid for 12 months from purchase, so yours expire September 15th, 2025."
- role: user
content: "What if I want to cancel my membership but keep the training sessions?"
- role: assistant
content: "You can downgrade to our Training-Only membership at $49 per month, which gives you facility access only when you have scheduled training sessions."
- role: user
content: "Hmm. What about freezing my membership instead?"
- role: assistant
content: "We offer membership freezes for $15 per month, minimum one month, maximum six months per year."
- role: user
content: "Have I used any freeze time this year?"
- role: assistant
content: "Yes, you froze your membership for two months - June and July when you were traveling."
- role: user
content: "Oh right, I forgot about that. So I have four months remaining?"
- role: assistant
content: "Correct, you could freeze for up to four more months in 2024."
- role: user
content: "What's my contract situation? Am I month-to-month or annual?"
- role: assistant
content: "You signed a 24-month agreement that ended in March 2023, so you've been month-to-month since then."
- role: user
content: "So I can cancel anytime?"
- role: assistant
content: "Yes, with 30 days notice. If you cancel today, your last day would be December 18th."
- role: user
content: "Would I get a refund for the partial month?"
- role: assistant
content: "No, we bill on the 1st of each month and don't prorate cancellations. Your December payment would cover through December 31st."
- role: user
content: "I see. What about my locker rental?"
- role: assistant
content: "You have a full-size locker, number 342, at $25 per month. That's billed separately and can be canceled independently."
- role: user
content: "And my husband's add-on membership?"
- role: assistant
content: "Mr. Kevin Chen is on your account as a spouse add-on for $89 per month."
- role: user
content: "Would he have to cancel too if I cancel?"
- role: assistant
content: "He could convert to his own primary membership at $189 per month, or cancel with you."
- role: user
content: "What about the kids' club access we have?"
- role: assistant
content: "The Kids' Club for your daughter Emma is $30 per month. She's registered as age 6, correct?"
- role: user
content: "Yes. Actually she just turned 7 last month."
- role: assistant
content: "I'll update that. Kids' Club is available until age 12, so she has several years left if you continue."
- role: user
content: "Are there any cancellation fees I should know about?"
- role: assistant
content: "Since you're month-to-month, there's no cancellation fee. Just the 30-day notice requirement."
- role: user
content: "What about that enrollment fee I paid when I joined?"
- role: assistant
content: "The $299 enrollment fee from 2021 was a one-time fee. It's non-refundable."
- role: user
content: "If I cancel and rejoin later?"
- role: assistant
content: "If you return within 90 days, we waive the enrollment fee. After 90 days, you'd pay the current enrollment fee, which is now $349."
- role: user
content: "It went up?"
- role: assistant
content: "Yes, it increased in January 2024. Your monthly rate is also grandfathered - new Diamond members pay $209 per month."
- role: user
content: "So I'm saving $20 a month with my current rate?"
- role: assistant
content: "Exactly. If you cancel and return, you'd pay the current rates."
- role: user
content: "What about my remaining personal training sessions if I cancel everything?"
- role: assistant
content: "You have three options: use them before canceling, transfer them to another member, or request a prorated refund."
- role: user
content: "What would the refund be?"
- role: assistant
content: "With 16 sessions remaining out of 24, that's two-thirds of $2,400, so $1,600 minus a 20% administrative fee."
- role: user
content: "So I'd get back $1,280?"
- role: assistant
content: "Correct. $1,600 minus $320 administrative fee equals $1,280."
- role: user
content: "That's a hefty fee. What about transferring to someone else?"
- role: assistant
content: "Transfers are free to immediate family members. There's a $100 transfer fee for non-family."
- role: user
content: "Can my husband use them?"
- role: assistant
content: "Yes, as your spouse, he could use them with no transfer fee. He'd just need to coordinate with James or another trainer."
- role: user
content: "Speaking of which, didn't I have some guest passes?"
- role: assistant
content: "Diamond members get 5 guest passes per month. This month you've used 2, so you have 3 remaining."
- role: user
content: "Do those roll over?"
- role: assistant
content: "No, guest passes reset on the 1st of each month. Use them or lose them."
- role: user
content: "What about that rewards program? I had some points."
- role: assistant
content: "You have 4,500 LifeRewards points. Those can be redeemed for merchandise, services, or month credits."
- role: user
content: "How much is a month credit?"
- role: assistant
content: "A free month requires 10,000 points. You could get a $50 credit for 2,500 points."
- role: user
content: "How do I earn points again?"
- role: assistant
content: "100 points per month for membership, 50 points per personal training session, and 25 points per group class attended."
- role: user
content: "So I've been earning steadily. What happens to points if I cancel?"
- role: assistant
content: "Points expire 60 days after membership cancellation. You'd want to redeem them first."
- role: user
content: "Good to know. Oh, what about that nutrition consultation I booked?"
- role: assistant
content: "You have a consultation with our dietitian Sarah scheduled for December 3rd at 2 PM."
- role: user
content: "Is that included in my membership?"
- role: assistant
content: "The initial consultation is included with Diamond. Follow-ups are $75 per session."
- role: user
content: "Did I prepay for any follow-ups?"
- role: assistant
content: "No, you haven't purchased any additional nutrition sessions."
- role: user
content: "What about my body composition scans?"
- role: assistant
content: "Diamond members get monthly InBody scans free. You did your last one October 28th."
- role: user
content: "Right. And I'm due for another one soon."
- role: assistant
content: "Yes, you can schedule one anytime after November 28th."
- role: user
content: "If I freeze instead of cancel, do I keep accumulating rewards points?"
- role: assistant
content: "No, point earning is suspended during freeze periods. But existing points don't expire."
- role: user
content: "And I can't use the facilities during a freeze?"
- role: assistant
content: "Correct, access is suspended except for the pro shop and café. You can't attend classes or use equipment."
- role: user
content: "What about my parking? I have that monthly parking pass."
- role: assistant
content: "The garage parking is $40 per month, billed on the 15th. That's a separate agreement with the garage, not through us."
- role: user
content: "Oh right, I'd need to cancel that separately with them."
- role: assistant
content: "Yes, they require 15 days notice for parking cancellation."
- role: user
content: "This is a lot to consider. Can you email me a summary?"
- role: assistant
content: "Of course. What's your email address?"
- role: user
content: "rchen.fitness@email.com"
- role: assistant
content: "I'll send a detailed summary of your membership, remaining services, and cancellation options."
- role: user
content: "Great. Oh, one more thing - didn't I refer my friend Lisa last month?"
- role: assistant
content: "Yes! Lisa Wang joined October 22nd. You received a $50 credit on your November billing."
- role: user
content: "And she got a discount too?"
- role: assistant
content: "She got her enrollment fee waived, saving $349."
- role: user
content: "Nice. Does she know I referred her?"
- role: assistant
content: "Yes, we inform new members who referred them when they use a referral code."
- role: user
content: "Perfect. I think I need a few days to decide what to do."
- role: assistant
content: "Take your time. Remember, if you decide to cancel, you need to do it by December 1st to avoid January billing."
- role: user
content: "I'll let you know by the end of the week. Thanks for all the information, Marcus."
- role: assistant
content: "You're welcome, Ms. Chen. Feel free to call back with any questions. Have a great rest of your day!"
user_question: "What was the exact cost breakdown for my personal training package including how much I'd get back if I requested a refund?"
evaluation_criteria: |
The agent should accurately recall:
- Original package: 24 sessions for $2,400
- Sessions used: 8
- Sessions remaining: 16
- Refund calculation: 16/24 * $2,400 = $1,600
- Administrative fee: 20% of $1,600 = $320
- Net refund: $1,280
The response should include all these specific numbers and the calculation method.
chapter3/user-memory-evaluation/test_cases/layer1/13_tax_preparation.yaml
+337
View File
@@ -0,0 +1,337 @@
test_id: layer1_13_tax_preparation
category: layer1
title: Tax Preparation Service - Deduction Details Retrieval
description: Test agent's ability to recall specific tax deductions, income sources, and filing details from a comprehensive tax preparation call
conversation_histories:
- conversation_id: tax_prep_001
timestamp: "2024-03-10 11:00:00"
metadata:
business: "H&R Block"
department: "Tax Services"
call_duration: "58 minutes"
messages:
- role: user
content: "Hi, I need help preparing my 2023 tax return."
- role: assistant
content: "Good morning! I'm Jennifer from H&R Block Tax Services. I'll help you prepare your 2023 return. Let's start with your basic information. Full name?"
- role: user
content: "Patricia Anne Sullivan."
- role: assistant
content: "Thank you, Ms. Sullivan. Your Social Security number?"
- role: user
content: "459-67-3821."
- role: assistant
content: "And your date of birth?"
- role: user
content: "September 8th, 1979."
- role: assistant
content: "Current address?"
- role: user
content: "5832 Willowbrook Lane, Denver, Colorado 80231."
- role: assistant
content: "Filing status - are you single, married filing jointly, married filing separately, or head of household?"
- role: user
content: "Married filing jointly."
- role: assistant
content: "Spouse's full name?"
- role: user
content: "Thomas Robert Sullivan."
- role: assistant
content: "His Social Security number?"
- role: user
content: "502-48-9156."
- role: assistant
content: "His date of birth?"
- role: user
content: "November 14th, 1977."
- role: assistant
content: "Any dependents?"
- role: user
content: "Yes, we have two children."
- role: assistant
content: "Their names and ages?"
- role: user
content: "Emily Rose Sullivan, she's 14, born June 3rd, 2009. And Jacob Thomas Sullivan, he's 11, born August 17th, 2012."
- role: assistant
content: "Social Security numbers for the children?"
- role: user
content: "Emily's is 506-22-8745 and Jacob's is 506-33-9821."
- role: assistant
content: "Perfect. Now let's discuss income. Starting with wages - do you have W-2s?"
- role: user
content: "Yes, I have one W-2 from Denver Regional Hospital where I work as a nurse practitioner."
- role: assistant
content: "What's in Box 1 - Wages, tips, other compensation?"
- role: user
content: "$94,500."
- role: assistant
content: "Box 2 - Federal income tax withheld?"
- role: user
content: "$18,725."
- role: assistant
content: "Box 3 - Social Security wages?"
- role: user
content: "$94,500."
- role: assistant
content: "Box 4 - Social Security tax withheld?"
- role: user
content: "$5,859."
- role: assistant
content: "Box 5 - Medicare wages?"
- role: user
content: "$94,500."
- role: assistant
content: "Box 6 - Medicare tax withheld?"
- role: user
content: "$1,370."
- role: assistant
content: "Box 12 - Any codes?"
- role: user
content: "Yes, Code D with $19,500."
- role: assistant
content: "That's your 401k contribution. Great. Does your spouse have W-2s?"
- role: user
content: "Yes, he works for Morrison Engineering. His Box 1 shows $118,000."
- role: assistant
content: "Federal tax withheld?"
- role: user
content: "$24,380."
- role: assistant
content: "Social Security tax?"
- role: user
content: "$7,316."
- role: assistant
content: "Medicare tax?"
- role: user
content: "$1,711."
- role: assistant
content: "Any 401k contributions?"
- role: user
content: "Yes, Box 12 Code D shows $22,500."
- role: assistant
content: "Excellent. Any other income? Interest, dividends, self-employment?"
- role: user
content: "We have interest from savings accounts. First National Bank sent a 1099-INT showing $1,845 in interest."
- role: assistant
content: "Any other interest income?"
- role: user
content: "Credit union savings showed $423."
- role: assistant
content: "Dividend income?"
- role: user
content: "Yes, from our Vanguard investment account. Ordinary dividends were $3,267 and qualified dividends were $2,890."
- role: assistant
content: "Any capital gains or losses?"
- role: user
content: "We sold some Apple stock. The 1099-B shows proceeds of $15,000 with a cost basis of $11,200."
- role: assistant
content: "So that's a $3,800 long-term capital gain. When did you originally purchase the stock?"
- role: user
content: "March 2019."
- role: assistant
content: "Good, definitely long-term. Any other investment sales?"
- role: user
content: "No, just that one."
- role: assistant
content: "Any rental income?"
- role: user
content: "No."
- role: assistant
content: "Self-employment or side business income?"
- role: user
content: "I do some freelance medical consulting. I made $8,500 total."
- role: assistant
content: "Did you receive 1099-NEC forms?"
- role: user
content: "Yes, one for $5,000 from MedConsult LLC and one for $3,500 from HealthAdvisors Inc."
- role: assistant
content: "Any business expenses against that income?"
- role: user
content: "I have a home office, computer, supplies, and some travel."
- role: assistant
content: "Let's detail those. Home office - what percentage of your home?"
- role: user
content: "My office is 150 square feet, and our house is 2,400 square feet total."
- role: assistant
content: "So 6.25% for home office. Your annual mortgage interest?"
- role: user
content: "The 1098 shows $14,800 in mortgage interest."
- role: assistant
content: "Property taxes?"
- role: user
content: "$4,200."
- role: assistant
content: "Utilities for the year?"
- role: user
content: "About $3,600 total."
- role: assistant
content: "Homeowners insurance?"
- role: user
content: "$1,800."
- role: assistant
content: "So your home office deduction would be 6.25% of those expenses, roughly $1,525. Other business expenses?"
- role: user
content: "New laptop for $1,899, office supplies about $340, and professional liability insurance $1,200."
- role: assistant
content: "Business mileage?"
- role: user
content: "I drove 1,847 miles for business consultations."
- role: assistant
content: "At 65.5 cents per mile for 2023, that's $1,210. Any other business expenses?"
- role: user
content: "Professional journal subscriptions, $450."
- role: assistant
content: "Total business expenses around $6,724. Now, let's discuss deductions. Will you itemize or take standard?"
- role: user
content: "What's better?"
- role: assistant
content: "Standard deduction for married filing jointly is $27,700 for 2023. Let's see if itemizing exceeds that. You mentioned $14,800 mortgage interest and $4,200 property taxes."
- role: user
content: "Right."
- role: assistant
content: "State income taxes paid?"
- role: user
content: "We paid $8,950 in Colorado state taxes."
- role: assistant
content: "Note that state and local taxes are capped at $10,000 for deduction. Charitable contributions?"
- role: user
content: "We donated $3,500 to our church, $1,000 to United Way, and $500 to the food bank."
- role: assistant
content: "So $5,000 in charitable contributions. That gives us $14,800 mortgage interest plus $10,000 SALT cap plus $5,000 charity equals $29,800."
- role: user
content: "So we should itemize?"
- role: assistant
content: "Yes, you'd save about $2,100 by itemizing. Any medical expenses?"
- role: user
content: "We had some out-of-pocket medical, about $3,200."
- role: assistant
content: "Medical expenses must exceed 7.5% of AGI to be deductible, so likely won't help. Any other deductions?"
- role: user
content: "We both contributed to IRAs."
- role: assistant
content: "Traditional or Roth?"
- role: user
content: "Traditional. I put in $6,500 and Tom put in $6,500."
- role: assistant
content: "With your income levels, the IRA deduction might be limited due to workplace retirement plan coverage. Let me calculate... Your AGI is too high for deductible traditional IRA contributions since you both have workplace plans."
- role: user
content: "Oh, I didn't realize that."
- role: assistant
content: "The contributions still grow tax-deferred, but no current deduction. Any education expenses?"
- role: user
content: "Emily takes piano lessons and Jacob is in travel soccer."
- role: assistant
content: "Those aren't deductible. Any college tuition?"
- role: user
content: "No, not yet."
- role: assistant
content: "Child care expenses for Jacob?"
- role: user
content: "He goes to after-school care. We paid $3,800 for the year."
- role: assistant
content: "That may qualify for the child care credit. Provider's name and tax ID?"
- role: user
content: "Sunshine After School Academy, EIN 84-1234567."
- role: assistant
content: "Perfect. Any energy-efficient home improvements?"
- role: user
content: "We installed solar panels in June."
- role: assistant
content: "Cost?"
- role: user
content: "$22,000 total."
- role: assistant
content: "That qualifies for a 30% federal tax credit, so $6,600 credit."
- role: assistant
content: "Any electric vehicle purchases?"
- role: user
content: "No, not this year."
- role: assistant
content: "Health Savings Account contributions?"
- role: user
content: "Yes! I forgot. We contributed $7,750 to our family HSA."
- role: assistant
content: "Was that through payroll deduction?"
- role: user
content: "No, we made direct contributions."
- role: assistant
content: "Great, that's fully deductible. Any cryptocurrency transactions?"
- role: user
content: "No."
- role: assistant
content: "Gambling winnings or losses?"
- role: user
content: "We won $800 on a trip to Vegas but probably lost more than that overall."
- role: assistant
content: "Do you have documentation of losses?"
- role: user
content: "Not really, no receipts."
- role: assistant
content: "Then we'll just report the $800 winnings. Did they issue a W-2G?"
- role: user
content: "No, it was under the reporting threshold."
- role: assistant
content: "Still needs to be reported. Any state tax refund from 2022?"
- role: user
content: "Yes, we got $1,245 back from Colorado."
- role: assistant
content: "Since you itemized in 2022, that's taxable income for 2023. Any estimated tax payments for 2023?"
- role: user
content: "We made quarterly payments of $1,500 each for the consulting income."
- role: assistant
content: "So $6,000 total in estimated payments. Let me calculate your tax... With all income and deductions, your taxable income is approximately $179,000."
- role: user
content: "What's our tax on that?"
- role: assistant
content: "Federal tax would be about $31,400 before credits. With the solar credit and child tax credits, down to about $21,600."
- role: user
content: "And we already paid?"
- role: assistant
content: "Through withholding and estimated payments, you've paid $49,105."
- role: user
content: "So we're getting a refund?"
- role: assistant
content: "Yes, approximately $27,500 federal refund."
- role: user
content: "That's more than I expected!"
- role: assistant
content: "The solar panel credit really helped. Want direct deposit?"
- role: user
content: "Yes, to our First National Bank checking."
- role: assistant
content: "Routing number?"
- role: user
content: "102000076."
- role: assistant
content: "Account number?"
- role: user
content: "4438829100."
- role: assistant
content: "Perfect. I'll prepare everything for your review and signature. Any questions?"
- role: user
content: "When will we get our refund?"
- role: assistant
content: "E-filed returns with direct deposit typically get refunds within 21 days."
- role: user
content: "Great! Thank you for all your help, Jennifer."
- role: assistant
content: "You're welcome! I'll email you the completed return for review within 24 hours."
user_question: "What was my total self-employment income and what were all the business expense categories I mentioned with their amounts?"
evaluation_criteria: |
The agent should accurately recall:
Self-employment income:
- MedConsult LLC: $5,000
- HealthAdvisors Inc: $3,500
- Total: $8,500
Business expenses:
- Home office deduction: $1,525 (6.25% of home expenses)
- Laptop: $1,899
- Office supplies: $340
- Professional liability insurance: $1,200
- Business mileage: $1,210 (1,847 miles at 65.5 cents)
- Professional journal subscriptions: $450
- Total expenses: $6,724
The response should include all income sources and expense categories with specific amounts.
chapter3/user-memory-evaluation/test_cases/layer1/14_cellphone_upgrade.yaml
+228
View File
@@ -0,0 +1,228 @@
test_id: layer1_14_cellphone_upgrade
category: layer1
title: Cell Phone Plan Upgrade - Device and Plan Details Retrieval
description: Test agent's ability to recall specific phone models, plan features, pricing, and trade-in values from a detailed upgrade discussion
conversation_histories:
- conversation_id: phone_upgrade_001
timestamp: "2024-11-25 13:45:00"
metadata:
business: "Verizon Wireless"
department: "Customer Upgrades"
call_duration: "56 minutes"
messages:
- role: user
content: "Hi, I'm interested in upgrading my phone and possibly changing my plan."
- role: assistant
content: "Hello! I'm Derek from Verizon Customer Upgrades. I'd be happy to help you explore your options. May I have your account number or phone number?"
- role: user
content: "My number is 415-555-7823."
- role: assistant
content: "Thank you. I see the account for... Mr. Jonathan Park?"
- role: user
content: "That's me."
- role: assistant
content: "Perfect. I see you've been with us since 2018. You currently have an iPhone 12 Pro on our Play More Unlimited plan at $90 per month."
- role: user
content: "Right. I'm thinking about getting the new iPhone 15 Pro Max."
- role: assistant
content: "Excellent choice. The 15 Pro Max comes in several storage options - 256GB at $1,199, 512GB at $1,399, or 1TB at $1,599."
- role: user
content: "What's my current phone's storage?"
- role: assistant
content: "Your iPhone 12 Pro is the 256GB model."
- role: user
content: "I'm always running out of space. What about the 512GB?"
- role: assistant
content: "The 512GB is $1,399 retail. With our device payment plan, that's $38.86 per month for 36 months."
- role: user
content: "Do you still do 24-month plans?"
- role: assistant
content: "Yes, 24 months would be $58.29 per month."
- role: user
content: "What about trade-in value for my current phone?"
- role: assistant
content: "Let me check... Is your iPhone 12 Pro in good condition? No cracks, water damage?"
- role: user
content: "It's in perfect condition. I've had it in a case with screen protector since day one."
- role: assistant
content: "Excellent. For a 256GB iPhone 12 Pro in good condition, we're offering $440 trade-in credit."
- role: user
content: "How does that apply?"
- role: assistant
content: "It's distributed as monthly credits over your payment term. For 36 months, that's $12.22 off per month, making your payment $26.64."
- role: user
content: "That's reasonable. What colors does the 15 Pro Max come in?"
- role: assistant
content: "Natural Titanium, Blue Titanium, White Titanium, and Black Titanium."
- role: user
content: "I like the Blue Titanium. Now what about plans? I feel like $90 is high."
- role: assistant
content: "Let me review our current plans. Your Play More includes unlimited premium data, 30GB mobile hotspot, and streaming perks."
- role: user
content: "What streaming perks do I have?"
- role: assistant
content: "Disney Bundle - Disney+, Hulu, and ESPN+, plus Apple Music."
- role: user
content: "I don't really use those. What are my other options?"
- role: assistant
content: "We have Start Unlimited at $70 per month - unlimited data but no hotspot or perks."
- role: user
content: "That might work. What about in between?"
- role: assistant
content: "Do More Unlimited is $80. It includes 50GB premium data and 25GB hotspot, but fewer entertainment perks."
- role: user
content: "What do you mean by premium data?"
- role: assistant
content: "After 50GB, your speeds may be temporarily slowed during network congestion. Start Unlimited doesn't have guaranteed premium data."
- role: user
content: "I use about 35GB per month usually."
- role: assistant
content: "Then Do More would ensure you never experience slowdowns. Start Unlimited might slow during busy times even under 35GB."
- role: user
content: "Let's go with Do More then. So $10 less per month."
- role: assistant
content: "Correct. From $90 to $80 monthly for the plan."
- role: user
content: "Any activation fees?"
- role: assistant
content: "There's a $35 upgrade fee, but I can offer you a $35 credit to offset that."
- role: user
content: "Great. What about insurance?"
- role: assistant
content: "Currently you have Total Mobile Protection at $15 per month. For the iPhone 15 Pro Max, it would be $17 per month."
- role: user
content: "What does that cover again?"
- role: assistant
content: "Unlimited cracked screen repairs for $29, device replacement for loss/theft/damage with $149 deductible for the 15 Pro Max."
- role: user
content: "I've never used it in three years."
- role: assistant
content: "You could switch to basic protection at $14 monthly - covers everything except loss and theft."
- role: user
content: "Actually, my credit card has phone protection. Can I cancel insurance?"
- role: assistant
content: "Absolutely. That saves you $17 per month. Just verify your card's coverage terms."
- role: user
content: "They cover up to $800 with a $50 deductible."
- role: assistant
content: "That's decent coverage. The 15 Pro Max is $1,399, so you'd have gap coverage risk of $599."
- role: user
content: "I'll take that risk. What about accessories?"
- role: assistant
content: "Any interest in cases, screen protectors, chargers?"
- role: user
content: "What's the best case for protection?"
- role: assistant
content: "The OtterBox Defender series at $64.99 or the Verizon exclusive Pelican Voyager at $59.99."
- role: user
content: "I'll take the Pelican. What about screen protectors?"
- role: assistant
content: "ZAGG Glass Elite is $44.99, or our Verizon tempered glass is $29.99."
- role: user
content: "The Verizon one is fine. Do I need a new charger?"
- role: assistant
content: "The phone comes with a USB-C cable but no power adapter. Do you have USB-C adapters?"
- role: user
content: "Just old USB-A ones."
- role: assistant
content: "The 20W Apple adapter is $29, or a third-party Anker 30W is $24.99."
- role: user
content: "Let's go with the Anker. Better value."
- role: assistant
content: "Smart choice. Anything else? Wireless charger, AirPods?"
- role: user
content: "No, I'm good. Can we review the total?"
- role: assistant
content: "iPhone 15 Pro Max 512GB Blue Titanium: $1,399, minus $440 trade-in = $959 financed."
- role: user
content: "Monthly payment?"
- role: assistant
content: "Over 36 months: $26.64 per month after trade-in credits."
- role: user
content: "And accessories?"
- role: assistant
content: "Pelican case $59.99, screen protector $29.99, Anker charger $24.99, total $114.97."
- role: user
content: "Can those be added to my monthly bill?"
- role: assistant
content: "Yes, accessories can be split over 6 months at $19.16 per month, no interest."
- role: user
content: "Perfect. And my plan change?"
- role: assistant
content: "Moving from Play More at $90 to Do More at $80, saving $10 monthly."
- role: user
content: "So my new monthly bill would be?"
- role: assistant
content: "$80 plan plus $26.64 phone payment, plus $19.16 for accessories for the first 6 months."
- role: user
content: "That's $125.80 for six months, then $106.64 after?"
- role: assistant
content: "Exactly right. Currently you're paying $90 for plan plus $15 insurance, so $105 total."
- role: user
content: "So basically the same after the accessories are paid off."
- role: assistant
content: "Yes, and you get a much better phone with double the storage."
- role: user
content: "When would the new phone arrive?"
- role: assistant
content: "The Blue Titanium 512GB is in stock. With free 2-day shipping, you'd have it by Wednesday."
- role: user
content: "Do I send my old phone back first?"
- role: assistant
content: "No, we include a prepaid trade-in kit. You have 30 days to send it after receiving the new phone."
- role: user
content: "What if I forget?"
- role: assistant
content: "You'd lose the trade-in credit and be charged the full $38.86 monthly instead of $26.64."
- role: user
content: "I'll set a reminder. Can I transfer my data at a store?"
- role: assistant
content: "Yes, any corporate store offers free data transfer. Or you can use iCloud backup yourself."
- role: user
content: "I'll do it myself. What about my current phone number?"
- role: assistant
content: "Your 415-555-7823 number stays the same. We'll include a SIM card, or you can use eSIM."
- role: user
content: "What's easier?"
- role: assistant
content: "eSIM - just scan a QR code we'll email you. No physical card needed."
- role: user
content: "Let's do eSIM then. Oh, what about international service?"
- role: assistant
content: "Do More includes Mexico and Canada. Other countries are $10 per day with TravelPass."
- role: user
content: "I'm going to Japan in March. How does that work?"
- role: assistant
content: "TravelPass charges $10 only on days you use your phone in Japan. Includes your plan's data and minutes."
- role: user
content: "Reasonable. Okay, let's proceed with the order."
- role: assistant
content: "Perfect! I'll process this now. You'll receive email confirmation within 10 minutes."
- role: user
content: "Great, thank you Derek!"
- role: assistant
content: "You're welcome! Your order number will be VZ-789456123. Is there anything else I can help with today?"
- role: user
content: "No, that covers everything. Thanks again!"
- role: assistant
content: "My pleasure, Mr. Park. Enjoy your new iPhone 15 Pro Max!"
user_question: "What were the exact model, storage, and color of the phone I ordered, and what was my total monthly cost breakdown?"
evaluation_criteria: |
The agent should accurately recall:
Phone ordered:
- Model: iPhone 15 Pro Max
- Storage: 512GB
- Color: Blue Titanium
- Retail price: $1,399
Monthly cost breakdown:
- Plan (Do More Unlimited): $80
- Phone payment after trade-in: $26.64 (36 months)
- Accessories (first 6 months): $19.16
- Total for first 6 months: $125.80
- Total after 6 months: $106.64
Trade-in credit: $440 for iPhone 12 Pro 256GB
The response should include all these specific details about the order.
chapter3/user-memory-evaluation/test_cases/layer1/15_college_enrollment.yaml
+247
View File
@@ -0,0 +1,247 @@
test_id: layer1_15_college_enrollment
category: layer1
title: College Enrollment Assistance - Course Registration Details
description: Test agent's ability to recall course selections, scheduling details, tuition costs, and financial aid information from enrollment call
conversation_histories:
- conversation_id: college_enroll_001
timestamp: "2024-07-15 10:00:00"
metadata:
business: "State University Admissions"
department: "Enrollment Services"
call_duration: "61 minutes"
messages:
- role: user
content: "Hi, I need help completing my enrollment for the fall semester."
- role: assistant
content: "Good morning! I'm Maria from State University Enrollment Services. I'll help you with your fall enrollment. May I have your student ID?"
- role: user
content: "It's SU-20240847."
- role: assistant
content: "Thank you. I see here... Ms. Ashley Rodriguez?"
- role: user
content: "Yes, that's me."
- role: assistant
content: "Welcome to State University, Ashley! I see you're an incoming freshman majoring in Computer Science."
- role: user
content: "That's right."
- role: assistant
content: "Have you completed your math placement exam?"
- role: user
content: "Yes, I took it last week."
- role: assistant
content: "Great! You scored 87, which places you directly into Calculus I. No remedial math needed."
- role: user
content: "Oh good! What about English placement?"
- role: assistant
content: "Your SAT scores exempt you from placement testing. You're cleared for English Composition 101."
- role: user
content: "Perfect. So what courses should I take first semester?"
- role: assistant
content: "For Computer Science majors, we recommend: Calculus I, Introduction to Computer Science, English Composition, and one general education course."
- role: user
content: "How many credits is that?"
- role: assistant
content: "Calculus I is 4 credits, Intro to CS is 4 credits, English is 3 credits. With a 3-credit gen ed, that's 14 credits."
- role: user
content: "Should I take more?"
- role: assistant
content: "First semester, we recommend 14-16 credits while you adjust to college. You could add another 3-credit course."
- role: user
content: "What gen ed options do I have?"
- role: assistant
content: "You need humanities, social sciences, and natural sciences. Any preference?"
- role: user
content: "I'm interested in psychology."
- role: assistant
content: "Perfect! Psychology 101 fulfills social sciences. Professor Williams teaches MWF at 10 AM or Professor Chen teaches Tuesday/Thursday at 2 PM."
- role: user
content: "I'm more of a morning person. Let's do MWF at 10."
- role: assistant
content: "Added Psychology 101 with Professor Williams, section 03, MWF 10:00-10:50 AM."
- role: user
content: "What about the Calc class?"
- role: assistant
content: "Calculus I has several options. Professor Johnson has MWF 8:00-9:15 AM, or Professor Kumar has MWF 1:00-2:15 PM."
- role: user
content: "8 AM is early but I'll take it. Better to get it done."
- role: assistant
content: "Registered for MAT-151 Calculus I, section 01, with Professor Johnson, MWF 8:00-9:15 AM, plus Thursday recitation."
- role: user
content: "Recitation?"
- role: assistant
content: "A smaller problem-solving session with a TA. Thursday sections are at 3:00 PM or 5:00 PM."
- role: user
content: "3 PM works better."
- role: assistant
content: "Added Thursday 3:00-3:50 PM recitation, section R04."
- role: user
content: "Now the Computer Science course?"
- role: assistant
content: "CS-101 Introduction to Computer Science. Professor Martinez teaches Tuesday/Thursday 9:30-10:45 AM with a 2-hour lab."
- role: user
content: "When are the labs?"
- role: assistant
content: "Lab options are Wednesday 2:00-4:00 PM or Friday 3:00-5:00 PM."
- role: user
content: "Wednesday at 2 PM is good."
- role: assistant
content: "Registered for CS-101, section 02, T/Th 9:30-10:45 AM, with Wednesday 2:00-4:00 PM lab, section L03."
- role: user
content: "And English?"
- role: assistant
content: "English 101 has many sections. There's T/Th 11:00 AM, MWF 2:00 PM, or T/Th 3:30 PM."
- role: user
content: "T/Th at 11 would work well after my CS class."
- role: assistant
content: "Perfect. ENG-101 with Professor Thompson, section 07, Tuesday/Thursday 11:00 AM-12:15 PM."
- role: user
content: "Should I add one more class to get to 15 or 16 credits?"
- role: assistant
content: "You could add a 1-credit freshman seminar or a 3-credit elective. What interests you?"
- role: user
content: "What's the freshman seminar about?"
- role: assistant
content: "FYE-100 'First Year Experience' - study skills, campus resources, career planning. Meets once weekly."
- role: user
content: "That sounds helpful. When does it meet?"
- role: assistant
content: "Mondays 3:00-3:50 PM or Wednesdays 12:00-12:50 PM."
- role: user
content: "Mondays at 3 PM works."
- role: assistant
content: "Added FYE-100, section 12, Mondays 3:00-3:50 PM with advisor Mr. Davis. That brings you to 15 credits."
- role: user
content: "Perfect. Now what about housing? I applied for campus dorms."
- role: assistant
content: "Let me check... You're assigned to Lincoln Hall, Room 347, with roommate Jennifer Kim."
- role: user
content: "Is that a double or triple?"
- role: assistant
content: "It's a double room. Lincoln Hall has suite-style rooms - two doubles sharing a bathroom."
- role: user
content: "So four people share one bathroom?"
- role: assistant
content: "Correct. Your suitemates in room 345 are Emma Wilson and Sarah Chen."
- role: user
content: "What floor is that?"
- role: assistant
content: "Third floor. Lincoln Hall has elevators and is co-ed by floor - third floor is all female."
- role: user
content: "Good. What about meal plans?"
- role: assistant
content: "Freshmen must choose from three plans: Unlimited at $2,850/semester, 14 meals/week at $2,450, or 10 meals/week at $2,150."
- role: user
content: "What's the difference?"
- role: assistant
content: "Unlimited includes all dining halls anytime. The others include that many meal swipes plus $200 dining dollars for cafés."
- role: user
content: "I'll probably eat breakfast in my room. Let's do 14 meals."
- role: assistant
content: "Good choice. That's lunch and dinner daily with flexibility. Selected 14-meal plan at $2,450."
- role: user
content: "Now about tuition - what do I owe?"
- role: assistant
content: "In-state tuition is $6,840 per semester for 12-18 credits. You're at 15 credits, so that's the flat rate."
- role: user
content: "Plus fees?"
- role: assistant
content: "Student fees are $782, technology fee $145, health services $210. Total fees: $1,137."
- role: user
content: "So tuition and fees together?"
- role: assistant
content: "$7,977 per semester."
- role: user
content: "Plus housing and meals?"
- role: assistant
content: "Lincoln Hall is $3,925 per semester. With the meal plan, housing and dining totals $6,375."
- role: user
content: "So my total semester bill is?"
- role: assistant
content: "$14,352 for fall semester."
- role: user
content: "I have financial aid. Can you see my package?"
- role: assistant
content: "Yes, you have: Federal Pell Grant $3,200, State Need Grant $2,500, and Merit Scholarship $4,000 per semester."
- role: user
content: "That's $9,700 in aid?"
- role: assistant
content: "Correct. That leaves $4,652 due for fall semester."
- role: user
content: "Can I do a payment plan?"
- role: assistant
content: "Yes, we offer a 5-month plan. August through December at $930.40 per month, no interest."
- role: user
content: "When's the first payment?"
- role: assistant
content: "August 15th, then the 15th of each month through December."
- role: user
content: "That's manageable. How do I set that up?"
- role: assistant
content: "I can enroll you now. There's a $35 enrollment fee for the payment plan."
- role: user
content: "That's fine. Use my checking account?"
- role: assistant
content: "Yes, we can do automatic bank drafts. I'll need your routing and account numbers."
- role: user
content: "Routing is 121000358, account is 5529847100."
- role: assistant
content: "Perfect. Payment plan established. You'll receive confirmation via email."
- role: user
content: "What about books?"
- role: assistant
content: "Average is $200-300 per class. For your five classes, budget about $1,200."
- role: user
content: "That's a lot! Any way to save?"
- role: assistant
content: "Check our bookstore's rental program, or buy used. Also, some professors use free online resources."
- role: user
content: "I'll look into rentals. When can I buy books?"
- role: assistant
content: "Book lists are posted August 1st. Classes start August 28th."
- role: user
content: "When do I move into the dorm?"
- role: assistant
content: "Freshman move-in is August 24th-25th. You're assigned August 24th, 10 AM - 12 PM time slot."
- role: user
content: "Is there orientation?"
- role: assistant
content: "Yes, mandatory orientation runs August 25th-27th. Includes campus tour, advisor meetings, and social events."
- role: user
content: "Do parents come to that?"
- role: assistant
content: "There's a separate parent orientation on August 25th from 2-5 PM."
- role: user
content: "Good, my mom wants to attend. What about parking?"
- role: assistant
content: "Freshmen can purchase parking permits for $450/semester for Lot G, about 10-minute walk to Lincoln Hall."
- role: user
content: "I won't have a car first semester."
- role: assistant
content: "That's common. We have good campus shuttle service and the city bus is free with student ID."
- role: user
content: "Perfect. I think that covers everything!"
- role: assistant
content: "Great! You're all set for fall. Your next steps are attending orientation and buying books. Any other questions?"
- role: user
content: "No, thank you so much for your help, Maria!"
- role: assistant
content: "You're welcome, Ashley! Have a wonderful freshman year at State University!"
user_question: "What was my complete course schedule including days, times, and professors, and what was my total semester cost after financial aid?"
evaluation_criteria: |
The agent should accurately recall:
Course Schedule:
1. MAT-151 Calculus I - Professor Johnson - MWF 8:00-9:15 AM, Thursday recitation 3:00-3:50 PM (4 credits)
2. CS-101 Intro to Computer Science - Professor Martinez - T/Th 9:30-10:45 AM, Wednesday lab 2:00-4:00 PM (4 credits)
3. PSY-101 Psychology - Professor Williams - MWF 10:00-10:50 AM (3 credits)
4. ENG-101 English Composition - Professor Thompson - T/Th 11:00 AM-12:15 PM (3 credits)
5. FYE-100 First Year Experience - Mr. Davis - Monday 3:00-3:50 PM (1 credit)
Total: 15 credits
Financial Breakdown:
- Total semester cost: $14,352
- Financial aid: $9,700 (Pell Grant $3,200 + State Grant $2,500 + Merit Scholarship $4,000)
- Amount due: $4,652
- Payment plan: $930.40/month for 5 months
The response should include all courses with specific meeting times and the complete financial calculation.
chapter3/user-memory-evaluation/test_cases/layer1/16_home_renovation.yaml
+244
View File
@@ -0,0 +1,244 @@
test_id: layer1_16_home_renovation
category: layer1
title: Home Renovation Quote - Detailed Cost Breakdown Retrieval
description: Test agent's ability to recall specific renovation items, materials, labor costs, and timeline from contractor consultation
conversation_histories:
- conversation_id: renovation_quote_001
timestamp: "2024-11-12 14:00:00"
metadata:
business: "Premier Home Renovations"
department: "Project Estimation"
call_duration: "54 minutes"
messages:
- role: user
content: "Hi, I got your number from a neighbor. I need a quote for renovating my kitchen and bathroom."
- role: assistant
content: "Good afternoon! I'm Tom from Premier Home Renovations. I'd be happy to provide you with a detailed quote. Let's start with your contact information."
- role: user
content: "I'm Linda Harrison. My address is 4521 Oakwood Drive, Seattle, Washington 98102."
- role: assistant
content: "Thank you, Ms. Harrison. And the best phone number to reach you?"
- role: user
content: "206-555-3847."
- role: assistant
content: "Perfect. Now, tell me about your kitchen. What's the current square footage?"
- role: user
content: "It's about 180 square feet."
- role: assistant
content: "And what type of renovation are you looking for? Full gut or cosmetic update?"
- role: user
content: "Full renovation. The kitchen is from the 1970s and needs everything replaced."
- role: assistant
content: "Understood. Let's go through each component. Starting with cabinets - what style interests you?"
- role: user
content: "I like shaker style cabinets in white."
- role: assistant
content: "Excellent choice. For 180 square feet, you'd need approximately 25 linear feet of cabinetry. Our mid-range shaker cabinets run $285 per linear foot."
- role: user
content: "So that would be?"
- role: assistant
content: "$7,125 for the cabinets. That includes soft-close hardware and crown molding."
- role: user
content: "What about installation?"
- role: assistant
content: "Cabinet installation is $125 per linear foot, so $3,125 for labor."
- role: user
content: "Okay. What about countertops?"
- role: assistant
content: "What material were you considering?"
- role: user
content: "I really want quartz."
- role: assistant
content: "Good choice. For your kitchen, you'd need about 45 square feet. Our quartz options range from $75 to $125 per square foot installed."
- role: user
content: "Let's go with something in the middle, around $95."
- role: assistant
content: "That would be $4,275 for quartz countertops, including fabrication and installation."
- role: user
content: "What about the backsplash?"
- role: assistant
content: "Subway tile is popular. Classic white 3x6 subway tile with installation runs about $18 per square foot."
- role: user
content: "How many square feet would I need?"
- role: assistant
content: "Typically 30 square feet for your kitchen size. That's $540 for backsplash."
- role: user
content: "Now appliances - I need all new ones."
- role: assistant
content: "Are you looking for standard or high-end appliances?"
- role: user
content: "Mid-range is fine. Stainless steel."
- role: assistant
content: "A good appliance package - refrigerator, range, dishwasher, and microwave - runs about $4,800."
- role: user
content: "Specific brands?"
- role: assistant
content: "We typically recommend Samsung or LG. Samsung package with French door fridge, gas range, dishwasher, and over-range microwave."
- role: user
content: "Sounds good. What about the sink?"
- role: assistant
content: "Undermount stainless steel sink, 33-inch double bowl, about $450. Faucet?"
- role: user
content: "Something with a pull-down sprayer."
- role: assistant
content: "Delta or Moen pull-down faucets run $325-400. Let's say $375."
- role: user
content: "What about flooring?"
- role: assistant
content: "What's there now?"
- role: user
content: "Old vinyl that needs to go."
- role: assistant
content: "Luxury vinyl plank is popular - waterproof and durable. About $4.50 per square foot installed."
- role: user
content: "For 180 square feet?"
- role: assistant
content: "That's $810 for flooring. Demo of old floor adds $360."
- role: user
content: "What about lighting?"
- role: assistant
content: "Recessed LED lights, about $185 per fixture installed. I'd recommend 6 for your space."
- role: user
content: "So $1,110 for lighting?"
- role: assistant
content: "Correct. Plus under-cabinet lighting for $450 if interested."
- role: user
content: "Yes, let's add that. Now what about electrical work?"
- role: assistant
content: "We'll need to add outlets for code compliance - probably 4 new outlets at $275 each."
- role: user
content: "That's $1,100 for electrical?"
- role: assistant
content: "Yes, plus moving any existing outlets for new appliance locations, about $400 more."
- role: user
content: "Okay. Let's talk about the bathroom now."
- role: assistant
content: "What size is the bathroom?"
- role: user
content: "It's a full bath, about 60 square feet."
- role: assistant
content: "And what needs updating?"
- role: user
content: "Everything - tub, toilet, vanity, flooring, the works."
- role: assistant
content: "Let's start with the tub. Standard tub/shower combo or something special?"
- role: user
content: "I'd like a walk-in shower, no tub."
- role: assistant
content: "A tiled walk-in shower with glass door runs about $4,500 complete."
- role: user
content: "That includes the tile?"
- role: assistant
content: "Yes, standard 12x12 porcelain tile. Fancier tile would be extra."
- role: user
content: "That's fine. What about the vanity?"
- role: assistant
content: "36-inch vanity with quartz top and undermount sink is about $1,450."
- role: user
content: "And installation?"
- role: assistant
content: "Installation adds $325, plus plumbing connections $275."
- role: user
content: "The toilet?"
- role: assistant
content: "Comfort height elongated toilet, about $385 for a Kohler, plus $200 installation."
- role: user
content: "Bathroom flooring?"
- role: assistant
content: "Porcelain tile, 12x24 planks are popular. $8 per square foot installed."
- role: user
content: "So 60 square feet would be?"
- role: assistant
content: "$480 for bathroom flooring."
- role: user
content: "What about bathroom lighting and fan?"
- role: assistant
content: "Vanity light fixture $225, exhaust fan with humidity sensor $285, both installed."
- role: user
content: "Medicine cabinet?"
- role: assistant
content: "Recessed medicine cabinet with mirror, about $420 installed."
- role: user
content: "Paint for both rooms?"
- role: assistant
content: "Kitchen and bathroom painting, about $780 total including ceiling."
- role: user
content: "What about permits?"
- role: assistant
content: "Permits for kitchen and bath renovation in Seattle run about $850."
- role: user
content: "Is there a design fee?"
- role: assistant
content: "Basic design is included. Custom design work is $1,500 but optional."
- role: user
content: "Let's skip custom design. What's the total so far?"
- role: assistant
content: "Let me calculate... Kitchen is $26,605 and bathroom is $8,795, plus permits $850. Total: $36,250."
- role: user
content: "That's higher than I expected. Any place to save?"
- role: assistant
content: "You could go with laminate countertops instead of quartz, saves about $2,500. Or standard cabinets instead of shaker, saves $1,800."
- role: user
content: "Let's keep the quartz but go with standard cabinets."
- role: assistant
content: "That brings cabinets from $7,125 to $5,325, total project to $34,450."
- role: user
content: "Better. What about timeline?"
- role: assistant
content: "Kitchen takes 3-4 weeks, bathroom 2 weeks. We can overlap some work, total about 5 weeks."
- role: user
content: "When could you start?"
- role: assistant
content: "We have an opening starting January 8th, 2025."
- role: user
content: "So we'd be done by mid-February?"
- role: assistant
content: "Yes, completed by February 12th, barring any unforeseen issues."
- role: user
content: "What's your payment schedule?"
- role: assistant
content: "30% deposit to schedule ($10,335), 40% at project midpoint ($13,780), 30% on completion ($10,335)."
- role: user
content: "Do you warranty your work?"
- role: assistant
content: "Yes, one-year warranty on labor, manufacturers' warranties on all materials and appliances."
- role: user
content: "What if we find problems when you open up the walls?"
- role: assistant
content: "We'd notify you immediately with photos and revised quote for any additional work needed."
- role: user
content: "Like what kinds of issues?"
- role: assistant
content: "Water damage, outdated wiring, plumbing problems. Change orders typically run $500-3,000 depending on the issue."
- role: user
content: "Good to know. Can you email me this quote?"
- role: assistant
content: "Of course. What's your email address?"
- role: user
content: "lharrison.home@email.com"
- role: assistant
content: "I'll send a detailed breakdown within 2 hours. The quote is valid for 30 days."
- role: user
content: "Perfect. Thank you for the thorough estimate, Tom."
- role: assistant
content: "You're welcome, Ms. Harrison. Feel free to call with any questions. We'd love to transform your kitchen and bathroom!"
user_question: "What was the total renovation cost after adjustments, and what were the three payment installments?"
evaluation_criteria: |
The agent should accurately recall:
Cost adjustments:
- Original total: $36,250
- Adjustment: Standard cabinets instead of shaker (saved $1,800)
- Final total: $34,450
Payment schedule (30%-40%-30%):
- Deposit (30%): $10,335
- Midpoint (40%): $13,780
- Completion (30%): $10,335
Timeline:
- Start date: January 8, 2025
- Completion: February 12, 2025
- Duration: 5 weeks
The response should include the final adjusted total and all three payment amounts.
chapter3/user-memory-evaluation/test_cases/layer1/17_veterinary_care.yaml
+218
View File
@@ -0,0 +1,218 @@
test_id: layer1_17_veterinary_care
category: layer1
title: Veterinary Care Plan - Pet Medical History and Treatment Details
description: Test agent's ability to recall pet information, medical conditions, treatment plans, and medication schedules from detailed vet consultation
conversation_histories:
- conversation_id: vet_consult_001
timestamp: "2024-11-08 15:30:00"
metadata:
business: "Companion Animal Hospital"
department: "Veterinary Services"
call_duration: "49 minutes"
messages:
- role: user
content: "Hi, I need to discuss my dog's recent health issues and treatment options."
- role: assistant
content: "Hello, this is Dr. Sarah Mitchell from Companion Animal Hospital. I'd be happy to discuss your pet's health. Can you provide your name and your pet's name?"
- role: user
content: "I'm Robert Thompson, and my dog is Max."
- role: assistant
content: "Thank you, Mr. Thompson. Let me pull up Max's file... I see Max is a 7-year-old Golden Retriever, 78 pounds, last seen October 15th?"
- role: user
content: "Yes, that's right. He's been having some issues since then."
- role: assistant
content: "What symptoms have you noticed?"
- role: user
content: "He's been limping on his right rear leg and seems less energetic."
- role: assistant
content: "How long has the limping been going on?"
- role: user
content: "About two weeks now. It started gradually."
- role: assistant
content: "Any specific incident that might have caused it? A fall, rough play?"
- role: user
content: "Not that I noticed. He just started favoring it after our regular walks."
- role: assistant
content: "I see in his history that Max has mild hip dysplasia. This could be related. When was his last X-ray?"
- role: user
content: "I think it was about 18 months ago when he was first diagnosed."
- role: assistant
content: "We should do new X-rays to check progression. Has he been taking any medications?"
- role: user
content: "He's been on glucosamine supplements, the ones you prescribed."
- role: assistant
content: "The Cosequin DS? What dosage?"
- role: user
content: "Two tablets daily with food."
- role: assistant
content: "Good. Has that been helping until recently?"
- role: user
content: "It seemed to, but not anymore."
- role: assistant
content: "We may need to add an anti-inflammatory. How's his appetite?"
- role: user
content: "Still good, eating his usual amount."
- role: assistant
content: "What food is he on?"
- role: user
content: "Hill's Science Diet Adult Large Breed, about 3.5 cups per day."
- role: assistant
content: "Split into how many meals?"
- role: user
content: "Two meals, morning and evening."
- role: assistant
content: "Perfect. Any vomiting or diarrhea?"
- role: user
content: "No, his digestion seems fine."
- role: assistant
content: "Good. Now, for the limping, I'd recommend starting with Rimadyl for pain and inflammation. Max weighs 78 pounds, so he'd need 75mg twice daily."
- role: user
content: "Are there side effects?"
- role: assistant
content: "Possible GI upset, decreased appetite, or lethargy. We'll do blood work first to check liver and kidney function."
- role: user
content: "When should we do that?"
- role: assistant
content: "Can you bring him in tomorrow morning? We do fasting blood work, so no food after 10 PM tonight."
- role: user
content: "Tomorrow's Saturday - are you open?"
- role: assistant
content: "Yes, we're open 8 AM to 2 PM Saturdays. How's 9 AM?"
- role: user
content: "That works. What tests will you run?"
- role: assistant
content: "Complete blood count, chemistry panel, and urinalysis. Total cost is $285."
- role: user
content: "And the X-rays?"
- role: assistant
content: "Hip and leg X-rays would be $420. We can do those same visit if blood work looks good."
- role: user
content: "So about $700 total for diagnostics?"
- role: assistant
content: "Yes, $705 plus the exam fee of $65, so $770 total for tomorrow."
- role: user
content: "Okay. What about long-term treatment?"
- role: assistant
content: "Depending on X-ray results, we might consider adequan injections for joint health."
- role: user
content: "What's that?"
- role: assistant
content: "It's a series of injections that help rebuild joint cartilage. Loading dose is twice weekly for 4 weeks, then monthly."
- role: user
content: "Cost?"
- role: assistant
content: "Each injection is $85. So $680 for the loading series, then $85 monthly maintenance."
- role: user
content: "That's expensive. Are there alternatives?"
- role: assistant
content: "We could try oral joint supplements first. Dasuquin Advanced is stronger than Cosequin, about $75 per month."
- role: user
content: "Let's see what the X-rays show first."
- role: assistant
content: "Absolutely. Now, is Max current on vaccinations?"
- role: user
content: "I think so. When were they due?"
- role: assistant
content: "His rabies is good until March 2025, but his DHPP is due now - expired October 30th."
- role: user
content: "Oh, I missed that. Can we do it tomorrow?"
- role: assistant
content: "Yes, DHPP vaccine is $42. Also, his heartworm test is due."
- role: user
content: "He's been on preventative, the Heartgard Plus."
- role: assistant
content: "Good, but we still need annual testing. That's $47."
- role: user
content: "So adding those to tomorrow's visit?"
- role: assistant
content: "Yes, new total would be $859 for everything tomorrow."
- role: user
content: "Fine. What about his weight? The vet mentioned he could lose a few pounds last time."
- role: assistant
content: "Yes, ideal weight for Max would be 72-73 pounds. He's about 5 pounds over."
- role: user
content: "How should I adjust his food?"
- role: assistant
content: "Reduce to 3 cups daily instead of 3.5, and limit treats to 10% of daily calories."
- role: user
content: "He loves his treats though."
- role: assistant
content: "Try baby carrots or green beans as low-calorie alternatives. Max likes vegetables?"
- role: user
content: "He'll eat anything!"
- role: assistant
content: "Perfect. The weight loss will also help his joints."
- role: user
content: "Should I change his exercise routine?"
- role: assistant
content: "With the limping, shorter more frequent walks are better than long ones. How long are your walks now?"
- role: user
content: "Usually 45 minutes twice a day."
- role: assistant
content: "Try 20 minutes three times daily instead. Less stress on joints."
- role: user
content: "What about swimming? He loves water."
- role: assistant
content: "Swimming is excellent for dogs with joint issues! Low impact but great exercise."
- role: user
content: "There's a dog pool nearby that's heated."
- role: assistant
content: "Perfect. Two or three swim sessions weekly would be ideal, 15-20 minutes each."
- role: user
content: "The pool charges $25 per session or $80 monthly unlimited."
- role: assistant
content: "The monthly unlimited would be cost-effective if you go twice weekly."
- role: user
content: "I'll look into that. What about supplements besides joint ones?"
- role: assistant
content: "Omega-3 fish oil helps with inflammation. 1000mg per 10 pounds body weight, so about 7000-8000mg daily for Max."
- role: user
content: "Any brand recommendations?"
- role: assistant
content: "Nordic Naturals or Grizzly Salmon Oil are good. About $30-40 per month."
- role: user
content: "Should I start that now?"
- role: assistant
content: "Yes, it's safe to start immediately. Give with food to avoid stomach upset."
- role: user
content: "What symptoms should I watch for that would need immediate attention?"
- role: assistant
content: "Sudden inability to walk, crying in pain, swelling of the leg, or loss of appetite with the medication."
- role: user
content: "And for tomorrow, just water is okay tonight?"
- role: assistant
content: "Yes, water is fine. Just no food after 10 PM for accurate blood work."
- role: user
content: "Should I bring anything?"
- role: assistant
content: "Just Max's leash and any current medications he's taking. We'll need about 2 hours for everything."
- role: user
content: "So I should plan to pick him up around 11?"
- role: assistant
content: "Yes, 11 AM pickup would be perfect. We'll call if we finish earlier."
- role: user
content: "Great. See you at 9 AM tomorrow."
- role: assistant
content: "Perfect. Max is in good hands, Mr. Thompson. We'll get to the bottom of his limping and get him comfortable again."
- role: user
content: "Thank you, Dr. Mitchell. I really appreciate the thorough explanation."
- role: assistant
content: "You're very welcome. Don't hesitate to call if you have any concerns before tomorrow."
user_question: "What was Max's current weight versus ideal weight, and what was the complete cost breakdown for tomorrow's visit?"
evaluation_criteria: |
The agent should accurately recall:
Weight information:
- Current weight: 78 pounds
- Ideal weight: 72-73 pounds
- Overweight by: approximately 5 pounds
Tomorrow's visit costs:
- Blood work (CBC, chemistry, urinalysis): $285
- X-rays (hip and leg): $420
- Exam fee: $65
- DHPP vaccine: $42
- Heartworm test: $47
- Total: $859
The response should include both the weight details and the complete itemized cost breakdown.
chapter3/user-memory-evaluation/test_cases/layer1/18_retirement_planning.yaml
+212
View File
@@ -0,0 +1,212 @@
test_id: layer1_18_retirement_planning
category: layer1
title: Retirement Account Consultation - Investment Portfolio Details
description: Test agent's ability to recall retirement account balances, investment allocations, contribution limits, and retirement projections
conversation_histories:
- conversation_id: retirement_plan_001
timestamp: "2024-10-28 11:00:00"
metadata:
business: "Fidelity Investments"
department: "Retirement Planning"
call_duration: "57 minutes"
messages:
- role: user
content: "Hi, I'd like to review my retirement accounts and maybe make some changes to my investments."
- role: assistant
content: "Good morning! I'm Michael from Fidelity Retirement Planning. I'll help you review your accounts. May I have your account number or Social Security number?"
- role: user
content: "My account number is FID-8827439."
- role: assistant
content: "Thank you. I have your account here... Ms. Catherine Lewis?"
- role: user
content: "Yes, that's me."
- role: assistant
content: "I see you have multiple accounts with us. A traditional IRA, a Roth IRA, and a rollover IRA from a previous 401k."
- role: user
content: "Right, plus my current 401k through work, though that's with Vanguard."
- role: assistant
content: "Let's review what you have with us first. Your traditional IRA has a current balance of $127,845."
- role: user
content: "How's that invested?"
- role: assistant
content: "60% in FXAIX - our S&P 500 index fund, 30% in FTIHX - international index, and 10% in FXNAX - bond index."
- role: user
content: "What about the Roth?"
- role: assistant
content: "Your Roth IRA balance is $73,420. It's invested more aggressively - 70% FZROX total market index, 20% FZILX international, 10% FNILX large cap."
- role: user
content: "And the rollover?"
- role: assistant
content: "The rollover IRA from your previous employer has $248,500. Currently in our Freedom 2045 target-date fund."
- role: user
content: "So my total with Fidelity is?"
- role: assistant
content: "Across all three accounts: $449,765."
- role: user
content: "How old am I in your records?"
- role: assistant
content: "You're 47, birth date showing May 15, 1977."
- role: user
content: "Right. So I have about 20 years until retirement."
- role: assistant
content: "If you retire at 67, yes. What's your target retirement age?"
- role: user
content: "I'd love to retire at 62 if possible."
- role: assistant
content: "That's 15 years. Let's look at projections. What's your current salary?"
- role: user
content: "$135,000 per year."
- role: assistant
content: "And how much are you contributing annually to retirement accounts?"
- role: user
content: "I max out my 401k at work, plus $7,000 to the Roth IRA."
- role: assistant
content: "So that's $23,000 to the 401k for 2024, plus $7,000 Roth, total $30,000 annually."
- role: user
content: "Right. My employer matches 6% too."
- role: assistant
content: "That's another $8,100 annually. Does your spouse have retirement accounts?"
- role: user
content: "Yes, my husband has his own 401k. He has about $380,000 saved."
- role: assistant
content: "Combined, you have approximately $830,000 currently. With continued contributions, let me calculate..."
- role: user
content: "What rate of return are you assuming?"
- role: assistant
content: "Let's use 7% average annual return, which is conservative but realistic."
- role: user
content: "Okay."
- role: assistant
content: "Contributing $38,100 annually, with 7% returns, you'd have approximately $2.4 million in 15 years."
- role: user
content: "Is that enough to retire at 62?"
- role: assistant
content: "Using the 4% rule, that would provide $96,000 annual income. What do you estimate needing in retirement?"
- role: user
content: "We'd like about $120,000 per year to maintain our lifestyle."
- role: assistant
content: "There's a gap of $24,000 annually. Will you have other income sources?"
- role: user
content: "Social Security eventually."
- role: assistant
content: "At 62, you can claim early but at reduced benefits. Full retirement age for you is 67."
- role: user
content: "What would I get at 62 versus 67?"
- role: assistant
content: "Based on your earnings, estimated $2,100 monthly at 62, or $2,950 at 67."
- role: user
content: "So $25,200 annually at 62."
- role: assistant
content: "Correct. That covers your gap. Your husband would have his Social Security too."
- role: user
content: "He's two years older, so he'd get his sooner."
- role: assistant
content: "Good point. Now, should we adjust your investment allocation?"
- role: user
content: "I'm wondering if I should be more conservative as I get closer to retirement."
- role: assistant
content: "Current overall allocation across your Fidelity accounts is roughly 65% stocks, 35% bonds. That's appropriate for your age."
- role: user
content: "Should I increase bonds?"
- role: assistant
content: "A common rule is 'age in bonds' - so 47% bonds for you. But with 15 years to retirement, you can stay aggressive."
- role: user
content: "What would you recommend?"
- role: assistant
content: "I'd suggest 60% stocks, 40% bonds now, gradually shifting to 40% stocks, 60% bonds by retirement."
- role: user
content: "How do I rebalance?"
- role: assistant
content: "We can set up automatic rebalancing quarterly. Or you can do it manually annually."
- role: user
content: "What's the cost difference?"
- role: assistant
content: "Automatic rebalancing is free. The index funds you own have expense ratios of 0.015% to 0.045%."
- role: user
content: "That's pretty low."
- role: assistant
content: "Yes, index funds keep costs minimal. Active funds would be 0.5% to 1% or higher."
- role: user
content: "Let's stick with index funds then. But maybe adjust the allocation?"
- role: assistant
content: "For the traditional IRA, I'd suggest: 50% FXAIX S&P 500, 20% FTIHX international, 30% FXNAX bonds."
- role: user
content: "And the Roth?"
- role: assistant
content: "Keep the Roth aggressive since it grows tax-free: 60% FZROX, 30% FZILX, 10% bonds."
- role: user
content: "What about the rollover in the target-date fund?"
- role: assistant
content: "The Freedom 2045 fund automatically adjusts allocation. Currently it's 58% stocks, 42% bonds."
- role: user
content: "Should I keep it there or pick individual funds?"
- role: assistant
content: "Target-date funds are convenient but slightly higher fees - 0.12% versus 0.015% for index funds."
- role: user
content: "That's not much difference."
- role: assistant
content: "On $248,500, it's about $250 per year extra in fees."
- role: user
content: "I'll leave it for convenience then."
- role: assistant
content: "Reasonable choice. Any questions about contribution limits for next year?"
- role: user
content: "What are the 2025 limits?"
- role: assistant
content: "401k limit increases to $23,500, IRA stays at $7,000. Catch-up contributions at age 50 add $7,500 and $1,000 respectively."
- role: user
content: "So when I turn 50, I can contribute how much total?"
- role: assistant
content: "$31,000 to 401k, $8,000 to IRA, total $39,000 personally."
- role: user
content: "That's a nice boost. Should I prioritize Roth or traditional?"
- role: assistant
content: "At your income level, traditional 401k gives you tax deductions now. Roth IRA for tax-free growth."
- role: user
content: "Can I do backdoor Roth conversions?"
- role: assistant
content: "Your income might be too high for direct Roth IRA contributions. What's your household AGI?"
- role: user
content: "About $245,000 combined."
- role: assistant
content: "You're over the $240,000 limit for married filing jointly. Backdoor Roth would work."
- role: user
content: "How does that work exactly?"
- role: assistant
content: "Contribute $7,000 to traditional IRA as non-deductible, then immediately convert to Roth."
- role: user
content: "Any tax implications?"
- role: assistant
content: "Only on any gains between contribution and conversion. If done quickly, minimal or zero tax."
- role: user
content: "I should do that going forward then."
- role: assistant
content: "Yes, it maintains your Roth contributions despite high income."
- role: user
content: "Perfect. Can you email me a summary of these recommendations?"
- role: assistant
content: "Absolutely. Email on file is clewis.finance@email.com?"
- role: user
content: "Yes, that's correct."
- role: assistant
content: "I'll send a detailed summary with allocation recommendations and projection charts within an hour."
- role: user
content: "Thank you, Michael. This was very helpful!"
- role: assistant
content: "You're welcome, Ms. Lewis. You're well on track for a comfortable retirement at 62!"
user_question: "What were my three Fidelity account balances and their current investment allocations?"
evaluation_criteria: |
The agent should accurately recall:
Account balances:
1. Traditional IRA: $127,845
2. Roth IRA: $73,420
3. Rollover IRA: $248,500
Total: $449,765
Current allocations:
- Traditional IRA: 60% FXAIX (S&P 500), 30% FTIHX (international), 10% FXNAX (bonds)
- Roth IRA: 70% FZROX (total market), 20% FZILX (international), 10% FNILX (large cap)
- Rollover IRA: 100% in Freedom 2045 target-date fund (currently 58% stocks, 42% bonds)
The response should include all three account balances and their specific fund allocations.
chapter3/user-memory-evaluation/test_cases/layer1/19_wedding_venue.yaml
+206
View File
@@ -0,0 +1,206 @@
test_id: layer1_19_wedding_venue
category: layer1
title: Wedding Venue Booking - Event Package Details and Pricing
description: Test agent's ability to recall specific wedding package details, guest counts, menu selections, and payment terms from venue consultation
conversation_histories:
- conversation_id: wedding_venue_001
timestamp: "2024-11-05 16:00:00"
metadata:
business: "Grand Oak Manor"
department: "Event Planning"
call_duration: "52 minutes"
messages:
- role: user
content: "Hi, I'm calling about booking your venue for my wedding next year."
- role: assistant
content: "Hello! This is Amanda from Grand Oak Manor. Congratulations on your engagement! I'd love to help you plan your special day. What's your name?"
- role: user
content: "I'm Jennifer Park, and my fiancé is Daniel Kim."
- role: assistant
content: "Lovely to meet you, Jennifer. When are you planning to get married?"
- role: user
content: "We're looking at September 14th, 2025."
- role: assistant
content: "Let me check... Yes, September 14th is available! It's a Sunday. Is that okay?"
- role: user
content: "Perfect, we specifically wanted a Sunday for the lower rates."
- role: assistant
content: "Smart choice! Sunday rates are 20% less than Saturday. How many guests are you expecting?"
- role: user
content: "About 150 people."
- role: assistant
content: "Wonderful size for our Ballroom. That holds up to 200 guests comfortably."
- role: user
content: "What does the venue rental include?"
- role: assistant
content: "The Ballroom rental is $4,500 for Sunday, includes 6 hours, tables, chairs, white linens, and setup/breakdown."
- role: user
content: "What about ceremony space?"
- role: assistant
content: "Our Garden Terrace for ceremonies is an additional $1,500, weather permitting, with indoor backup in the Conservatory."
- role: user
content: "So $6,000 total for spaces?"
- role: assistant
content: "Correct. Now, do you want us to handle catering?"
- role: user
content: "Yes, we'd like to keep everything in one place."
- role: assistant
content: "Perfect. For 150 guests, our packages range from $95 to $165 per person."
- role: user
content: "What does the middle package include?"
- role: assistant
content: "Our Signature Package at $125 per person includes cocktail hour with 5 appetizers, three-course plated dinner, and wedding cake."
- role: user
content: "What appetizer options?"
- role: assistant
content: "You choose 5 from our list: shrimp cocktail, bruschetta, stuffed mushrooms, cheese display, crudités, mini quiches, spring rolls, or meatballs."
- role: user
content: "And for the main dinner?"
- role: assistant
content: "Guests choose between two entrées you pre-select. Popular combinations are filet mignon with salmon, or chicken marsala with vegetarian lasagna."
- role: user
content: "Let's go with the filet and salmon option."
- role: assistant
content: "Excellent choices. The filet is 8oz with garlic mashed potatoes, the salmon is herb-crusted with rice pilaf. Both include seasonal vegetables."
- role: user
content: "What about bar service?"
- role: assistant
content: "Bar packages are separate. Premium open bar is $45 per person, beer and wine only is $28, or cash bar is free setup with guest purchases."
- role: user
content: "We'll do beer and wine. So that's $28 per person on top of the $125?"
- role: assistant
content: "Yes, total $153 per person for food and beverage, which is $22,950 for 150 guests."
- role: user
content: "Plus the $6,000 venue fee?"
- role: assistant
content: "Correct, running total of $28,950. There's also 8.5% sales tax and 22% service charge on food and beverage."
- role: user
content: "How much does that add?"
- role: assistant
content: "Tax is $1,951, service charge is $5,049, bringing your estimated total to $35,950."
- role: user
content: "That's higher than I expected. Any way to reduce it?"
- role: assistant
content: "You could do buffet instead of plated service, that's $108 per person, saves $17 per guest."
- role: user
content: "How much would that save total?"
- role: assistant
content: "That would save $2,550 on food, plus reduce tax and service charges. New total would be about $33,000."
- role: user
content: "Let's stick with plated service but maybe adjust appetizers?"
- role: assistant
content: "We could do 3 appetizers instead of 5, saves $8 per person, $1,200 total."
- role: user
content: "That helps. What about the cake?"
- role: assistant
content: "Included cake serves 150, three tiers, choice of vanilla, chocolate, or red velvet with buttercream."
- role: user
content: "Can we do different flavors per tier?"
- role: assistant
content: "Absolutely! Many couples do vanilla, chocolate, and red velvet, one per tier."
- role: user
content: "Perfect. What about decorations?"
- role: assistant
content: "Basic centerpieces are $35 each, premium are $75. You'd need 15 tables for 150 guests."
- role: user
content: "So $525 for basic centerpieces?"
- role: assistant
content: "Correct. Our florist can also do bridal bouquet for $250, bridesmaids for $75 each."
- role: user
content: "I have 5 bridesmaids, so that would be $625 total for bouquets?"
- role: assistant
content: "Yes, $250 for yours plus $375 for bridesmaids."
- role: user
content: "What about music and DJ?"
- role: assistant
content: "Our preferred DJ is $1,200 for ceremony and reception, includes all equipment and lighting."
- role: user
content: "And photography?"
- role: assistant
content: "We partner with several photographers. Packages range from $2,500 to $5,000 for 8 hours coverage."
- role: user
content: "Let's budget $3,500 for photography."
- role: assistant
content: "Good choice. That typically includes two photographers and edited digital images."
- role: user
content: "So with everything we've discussed?"
- role: assistant
content: "Venue $6,000, food/beverage for 150 at $145 per person ($125 food + $28 bar - $8 appetizer reduction) is $21,750..."
- role: user
content: "Wait, I want to keep the 5 appetizers. Let's not reduce those."
- role: assistant
content: "Okay, back to $153 per person, that's $22,950 for food and beverage."
- role: user
content: "Plus tax and service?"
- role: assistant
content: "Right, food/beverage $22,950, tax $1,951, service $5,049, venue $6,000, total $35,950."
- role: user
content: "Then add decorations, DJ, and photography?"
- role: assistant
content: "Centerpieces $525, bouquets $625, DJ $1,200, photography $3,500, adds $5,850."
- role: user
content: "So grand total around $41,800?"
- role: assistant
content: "Yes, $41,800 for everything we've discussed."
- role: user
content: "What's your payment schedule?"
- role: assistant
content: "$5,000 deposit to hold the date, 50% due 6 months before, final balance 2 weeks before."
- role: user
content: "So $5,000 now, how much in March 2025?"
- role: assistant
content: "50% of remaining balance after deposit. $41,800 minus $5,000 is $36,800, half is $18,400 due March 14th."
- role: user
content: "And the rest two weeks before?"
- role: assistant
content: "Yes, final $18,400 due August 31st, 2025."
- role: user
content: "Is the deposit refundable?"
- role: assistant
content: "Refundable if cancelled more than 12 months out. After that, it becomes non-refundable."
- role: user
content: "So I have until September 2024 to cancel with refund?"
- role: assistant
content: "Correct, until September 14th, 2024."
- role: user
content: "What about tastings?"
- role: assistant
content: "Complimentary tasting for up to 4 people included once you book, scheduled 3-4 months before your wedding."
- role: user
content: "When would we need to finalize menu choices?"
- role: assistant
content: "Menu selections due 60 days before, final headcount 14 days before."
- role: user
content: "Perfect. Can you email me this quote?"
- role: assistant
content: "Of course! What's your email?"
- role: user
content: "jpark.wedding@email.com"
- role: assistant
content: "I'll send a detailed proposal within 2 hours. It's valid for 14 days."
- role: user
content: "Wonderful. Thank you so much, Amanda!"
- role: assistant
content: "My pleasure, Jennifer! I hope we can be part of your special day. Feel free to call with any questions!"
user_question: "What was the per-person cost breakdown and the three payment installments for my wedding?"
evaluation_criteria: |
The agent should accurately recall:
Per-person cost ($153 total):
- Food (Signature Package): $125
- Beer and wine bar: $28
- Total: $153 per person
For 150 guests: $22,950
Payment schedule:
- Deposit (upon booking): $5,000
- 50% of remaining (March 14, 2025): $18,400
- Final balance (August 31, 2025): $18,400
- Total: $41,800
Additional charges on food/beverage:
- Sales tax (8.5%): $1,951
- Service charge (22%): $5,049
The response should include the per-person breakdown and all three payment amounts with dates.
chapter3/user-memory-evaluation/test_cases/layer1/20_daycare_enrollment.yaml
+219
View File
@@ -0,0 +1,219 @@
test_id: layer1_20_daycare_enrollment
category: layer1
title: Daycare Enrollment Process - Childcare Schedule and Fee Structure
description: Test agent's ability to recall child information, care schedule, tuition rates, and enrollment requirements from daycare registration call
conversation_histories:
- conversation_id: daycare_enroll_001
timestamp: "2024-11-15 10:30:00"
metadata:
business: "Bright Beginnings Learning Center"
department: "Enrollment Services"
call_duration: "48 minutes"
messages:
- role: user
content: "Hi, I'm looking for daycare for my twin daughters starting in January."
- role: assistant
content: "Good morning! I'm Sharon from Bright Beginnings Learning Center. I'd be happy to help you with enrollment. How old are your daughters?"
- role: user
content: "They just turned 3 in October."
- role: assistant
content: "Perfect age for our Preschool 1 program! What are their names?"
- role: user
content: "Emma and Olivia Mitchell."
- role: assistant
content: "Beautiful names! And what's your name?"
- role: user
content: "I'm Barbara Mitchell."
- role: assistant
content: "Thank you, Ms. Mitchell. When exactly would you need care to start?"
- role: user
content: "January 6th, 2025, when I return from maternity leave."
- role: assistant
content: "Congratulations on your new baby! What days and hours do you need?"
- role: user
content: "Monday through Friday, probably 7:30 AM to 5:30 PM."
- role: assistant
content: "That's our standard full-time schedule. Let me check availability... Yes, we have two spots in our Butterfly classroom for 3-year-olds."
- role: user
content: "They'd be in the same classroom?"
- role: assistant
content: "Yes, we keep twins together unless parents request otherwise. The Butterfly room has 12 children with 2 teachers."
- role: user
content: "What's the teacher-child ratio?"
- role: assistant
content: "For 3-year-olds, it's 1:6, so two teachers for twelve children."
- role: user
content: "That's good. What's the tuition?"
- role: assistant
content: "Full-time for 3-year-olds is $325 per week per child."
- role: user
content: "So $650 weekly for both?"
- role: assistant
content: "Actually, we offer a 10% sibling discount on the second child. So $325 for Emma and $292.50 for Olivia, total $617.50 weekly."
- role: user
content: "That helps. How do you bill?"
- role: assistant
content: "Weekly tuition is due Monday mornings. We offer automatic withdrawal from checking or credit card."
- role: user
content: "What about registration fees?"
- role: assistant
content: "There's a one-time registration fee of $150 per child, so $300 for both."
- role: user
content: "Is that due immediately?"
- role: assistant
content: "Yes, registration fees hold their spots. Plus first week's tuition is due two weeks before start date."
- role: user
content: "So I'd pay $300 registration now, and $617.50 on December 23rd?"
- role: assistant
content: "Exactly right."
- role: user
content: "What about supplies?"
- role: assistant
content: "Annual supply fee is $200 per child, covers art materials, educational supplies, and field trips."
- role: user
content: "When is that due?"
- role: assistant
content: "Supply fee is due with first tuition payment, so December 23rd. That's another $400 for both girls."
- role: user
content: "What's included in the tuition?"
- role: assistant
content: "Breakfast, lunch, afternoon snack, all activities, and basic field trips. Special trips might have small fees."
- role: user
content: "What time is breakfast?"
- role: assistant
content: "Breakfast is 8:00-8:30 AM. If they arrive after 8:30, they should eat at home."
- role: user
content: "They'll be there by 7:30, so that's perfect. What about naps?"
- role: assistant
content: "Nap time is 12:30-2:30 PM. Do they still nap?"
- role: user
content: "Emma does, but Olivia usually just rests quietly."
- role: assistant
content: "That's fine. Non-nappers have quiet activities like puzzles or books on their cots."
- role: user
content: "What do they need to bring?"
- role: assistant
content: "Complete change of clothes, nap blanket, and small pillow if desired. We provide cots."
- role: user
content: "Are they potty trained required?"
- role: assistant
content: "For the 3-year-old room, we expect them to be mostly trained but accidents are normal. Are they trained?"
- role: user
content: "Emma is fully trained, Olivia has occasional accidents."
- role: assistant
content: "Very normal at this age. Just send extra clothes. We help with bathroom reminders."
- role: user
content: "What's the daily schedule like?"
- role: assistant
content: "7:30-8:00 free play, 8:00-8:30 breakfast, 8:30-9:00 circle time, 9:00-10:00 learning centers, 10:00-10:30 snack and story, 10:30-11:30 outdoor play..."
- role: user
content: "And afternoons?"
- role: assistant
content: "11:30-12:00 lunch, 12:00-12:30 bathroom and prep for nap, 12:30-2:30 nap/rest, 2:30-3:00 snack, 3:00-4:00 art or music, 4:00-5:00 outdoor play, 5:00-5:30 quiet activities and pickup."
- role: user
content: "What curriculum do you follow?"
- role: assistant
content: "We use Creative Curriculum, focusing on social-emotional development, pre-math, pre-literacy, and science exploration."
- role: user
content: "Do they learn letters and numbers?"
- role: assistant
content: "Yes, through play-based learning. By 4, most know uppercase letters and numbers 1-10."
- role: user
content: "What about sick policies?"
- role: assistant
content: "Children must be fever-free for 24 hours without medication before returning. Same for vomiting or diarrhea."
- role: user
content: "What if I need to pick them up late?"
- role: assistant
content: "We're open until 6:00 PM. After 5:30 PM is $1 per minute per child late fee."
- role: user
content: "So if I'm 10 minutes late?"
- role: assistant
content: "That would be $20 total, $10 per child."
- role: user
content: "What about vacation credits?"
- role: assistant
content: "After 6 months enrollment, you get one week vacation credit per year at 50% tuition."
- role: user
content: "So I'd pay half tuition for vacation week?"
- role: assistant
content: "Correct, $308.75 instead of $617.50 for that week."
- role: user
content: "What about holidays?"
- role: assistant
content: "We're closed major holidays - New Year's, Memorial Day, July 4th, Labor Day, Thanksgiving and day after, Christmas Eve and Christmas."
- role: user
content: "Do we still pay for those weeks?"
- role: assistant
content: "Yes, tuition remains the same for holiday weeks. It's calculated into the weekly rate."
- role: user
content: "What forms do I need for enrollment?"
- role: assistant
content: "Birth certificates, immunization records, emergency contacts, pediatrician info, and any allergy or medical information."
- role: user
content: "Emma has a peanut allergy."
- role: assistant
content: "We're a nut-free facility. You'll need an allergy action plan from your pediatrician and we'll keep Benadryl or EpiPen as prescribed."
- role: user
content: "That's reassuring. What about developmental concerns?"
- role: assistant
content: "Do either have any developmental delays or IEP services?"
- role: user
content: "Olivia has mild speech delay and gets speech therapy once a week."
- role: assistant
content: "We can accommodate therapy visits. The therapist can work with her here during the day."
- role: user
content: "That would be wonderful. Currently it's Wednesdays at 10 AM."
- role: assistant
content: "Perfect timing during learning centers. We have a quiet room they can use."
- role: user
content: "What about parent communication?"
- role: assistant
content: "Daily reports through our app - meals, naps, activities, mood. Plus monthly newsletters and parent-teacher conferences twice yearly."
- role: user
content: "When are conferences?"
- role: assistant
content: "October and March, but you can request meetings anytime."
- role: user
content: "Can I tour the facility?"
- role: assistant
content: "Absolutely! Can you come tomorrow at 10 AM or 4 PM?"
- role: user
content: "4 PM works better."
- role: assistant
content: "Great, I'll schedule you for tomorrow, November 16th at 4 PM. Bring the girls if you'd like."
- role: user
content: "I will. Should I bring the paperwork then?"
- role: assistant
content: "If you decide to enroll, yes. Or I can email the forms now to review."
- role: user
content: "Please email them to bmitchell@email.com"
- role: assistant
content: "I'll send those within the hour. To summarize: $300 registration to hold spots, then $1,017.50 due December 23rd for first week plus supplies."
- role: user
content: "And weekly tuition of $617.50 after that?"
- role: assistant
content: "Correct, starting January 13th for the second week."
- role: user
content: "Perfect. See you tomorrow at 4!"
- role: assistant
content: "Looking forward to meeting you and Emma and Olivia tomorrow, Ms. Mitchell!"
user_question: "What was the complete weekly tuition cost for both twins and what initial payments are required before they start?"
evaluation_criteria: |
The agent should accurately recall:
Weekly tuition:
- Emma: $325 per week (full price)
- Olivia: $292.50 per week (10% sibling discount)
- Total weekly: $617.50
Initial payments required:
- Registration fee: $300 ($150 per child) - due upon enrollment
- First week's tuition: $617.50 - due December 23, 2024
- Supply fee: $400 ($200 per child) - due December 23, 2024
- Total due December 23: $1,017.50
Start date: January 6, 2025
Schedule: Monday-Friday, 7:30 AM - 5:30 PM
The response should include both the weekly tuition breakdown and all initial payment amounts with their due dates.