Hi Arya,
By "debug symbols", I assume you are referring to the -g compiler option. So you are saying by throwing -g to either your application or to the DSPLINK library, you see a performance decrease.
According to table 3.7 in the Compiler manual: http://www.ti.com/lit/ug/spru187v/spru187v.pdf, there should not be any impact on optimization level. This may be a good question to run by the compiler forum. Only other thought I have is it may have somehow changed the memory placement of the various sections. You may want to compare the map files before and after applying the -g option to see if there are any differences. If the data is placed differently it could have an impact on the areas that are still being cached.
Best regards,
Vincent