The person on the other end of the line is likely to have very sensitive information -- your account number, your Social Security number, the types of accounts you have (and) the way you use your accounts.
If you've given them your credit-card number and/or bank-account number, you should watch those accounts very carefully.